{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-07T02:19:02.272939Z",
     "start_time": "2019-04-07T02:19:02.268259Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "***\n",
    "***\n",
    "\n",
    "# Introduction to Pytorch\n",
    "\n",
    "\n",
    "***\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "<img src= 'img/neuralnetwork/softmax-regression-scalargraph.png' width= \"500px\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# WHAT IS PYTORCH?\n",
    "\n",
    "It’s a Python-based scientific computing package targeted at two sets of audiences:\n",
    "\n",
    "- A replacement for NumPy to use the power of GPUs\n",
    "- a deep learning research platform that provides maximum flexibility and speed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "https://towardsdatascience.com/understanding-pytorch-with-an-example-a-step-by-step-tutorial-81fc5f8c4e8e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "PyTorch is the fastest growing Deep Learning framework\n",
    "- it is also used by Fast.ai in its MOOC, Deep Learning for Coders and its library.\n",
    "- PyTorch is also very pythonic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "PyTorch makes it much easier and more intuitive to build a Deep Learning model in Python\n",
    "- autograd, \n",
    "- dynamic computation graph, \n",
    "- model classes and more\n",
    "- avoid some common pitfalls and errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:35:58.186077Z",
     "start_time": "2019-06-19T08:35:57.462393Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from torch.autograd import Variable\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:22:58.131261Z",
     "start_time": "2019-06-19T11:22:58.124282Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Data Generation\n",
    "np.random.seed(42)\n",
    "x = np.random.rand(100, 1)\n",
    "y=1+2*x+.1*np.random.randn(100,1)\n",
    "# Shuffles the indices\n",
    "idx = np.arange(100)\n",
    "np.random.shuffle(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:35:59.133229Z",
     "start_time": "2019-06-19T08:35:59.125916Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Uses first 80 random indices for train\n",
    "train_idx = idx[:80]\n",
    "# Uses the remaining indices for validation\n",
    "val_idx = idx[80:]\n",
    "\n",
    "# Generates train and validation sets\n",
    "x_train, y_train = x[train_idx], y[train_idx]\n",
    "x_val, y_val = x[val_idx], y[val_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:36:00.364054Z",
     "start_time": "2019-06-19T08:36:00.096525Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEXCAYAAAC6baP3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xmc1nW5//HXNQswIDgqg8ooiFFgQUpimuZBPRqQZYiieVxDRTslHjWU0jxw4gS5pYfz08pUREXcgEDDJUGM7SgCWoYIiQLDmgISDDDL9fvjnhnuubm378y9zdzv5+MxD7y/6/WF8Xvdn93cHRERkWQVZDsAERFpWZQ4REQkECUOEREJRIlDREQCUeIQEZFAlDhERCQQJQ4REQlEiUNERAJR4hARkUCKsh1AOnTu3NmPOeaYbIchItKivPPOO/9w97JEx7XKxHHMMcewZMmSbIchItKimNknyRynqioREQlEiUNERAJR4hARkUCUOEREJBAlDhERCaRV9qoSEck3M5ZVcPcrK9mwvZKupSWMGtiLIf3K03KvvE0cn3/+OVu2bKGqqirboUiOKi4upkuXLnTq1CnboYjENWNZBT+d9hcqq2oAqNheyU+n/QUgLckjLxPH559/zubNmykvL6ekpAQzy3ZIkmPcncrKSioqKgCUPCSn3f3KyoakUa+yqoa7X1mZlsSRl20cW7Zsoby8nPbt2ytpSFRmRvv27SkvL2fLli3ZDkckrg3bKwNtb668TBxVVVWUlJRkOwxpAUpKSlSdKTmva2n091ms7c2VE4nDzArM7DUz+9DMVprZwIj9fczsXTP7xMwmmlmz41ZJQ5Kh3xNpCUYN7EVJcWGjbSXFhYwa2Cst98uJxAE4cIW7fwm4EfjviP0PAqOBY4GvAudlNjwRkdw1pF8544f2pby0BAPKS0sYP7Rv2npV5UTi8JCNdR+7A+/W7zOzMqCHu8929xrgKWBQFsJsFY4//niGDh0a6Jx77rmHzp0788knSc1/lhaTJk3CzNi0aVPWYhDJZUP6lbNg9FmsmXAuC0aflbakATnUq8rMbgVuA7YC4VVVRwFrwz6vB86Ncv4IYARAt27d0hdoDtmwYQMdOnTg4IMPTvqc+fPnU1xcHOg+N998M8OHD+fQQw8NGmJWuTsrV66kd+/e2Q5FJKUyOWYjmpwocQC4+13ufhjwM+AV21+53AaoDTu0FqiJcv7v3L2/u/cvK0s4nXyLt3DhQo466ihWrFgR6LyOHTvSrl27QOcUFBS0uKQBcP311zNkyJBshyGSUvVjNiq2V+LsH7MxY1lFxmLImcRRz92nAQcBh9Vt2giEp9KjgHWZjivX7Nu3D3fPdhg5be/evdkOQSTl4o3ZyJScSBxmdqyZHVH3398A9rj7PwDcfS2wy8zOMLNC4HLguexF29iMZRWcNmEOPUa/xGkT5mQk61911VWceeaZAHzjG9/AzHjjjTcAeOCBBzjhhBM46KCD6Nq1KzfeeCP79u1rOLd3795cddVVDZ/POOMMBg0axNSpU+nVqxcdOnRgwIABrFq1quGYCRMmNOpdVN/esHjxYgYNGkSHDh3o0aMHkyZNahTnpk2b+Ld/+zdKS0vp2LEjV1xxBaNGjUpY4tm9ezcjR47k8MMPp6SkhEGDBjUMxKu3Y8cObrnlFnr27ElJSQk9e/bk4YcfbthvZjz++OOsXLkSM6N+RcgNGzZw7bXX0r17d9q3b0+fPn34wx/+kPDvXCRXZHrMRjQ5kTiAUuBNM/s7cC9wsZmdb2Y/qdt/JTAR+Bh4093nZyfMxrJVZLzrrrt48sknAXjuuedYtWoVJ598ciimGTO49dZbmT9/Pr/4xS+YOHEiDz74YNzrLV++nGeffZbJkyczdepUVqxYwfDhwxPGccMNNzBixAjmzZtHnz59uOaaa1i5MvStp7a2lnPPPZdFixbx6KOPMm/ePLp06cLEiRMTXnfYsGFMmTKFu+++m0WLFnH22Wfzy1/+stExq1evZt26dTzwwAMsXryYwYMHc91117Fs2TIAVq1axdChQzn22GNZtWoV8+bNA+Ctt96ioKCAhx9+mIULF3LcccdxySWXsHHjxgPiEMlFmR6zEZW7t7qfE0880eP529/+Fnd/sk4d/7p3v+3FA35OHf96Sq4fz9y5cx3wRYsWxT3u61//up9//vkNn3v16uVXXnllw+cBAwZ4WVmZ7969u2HbmDFjHGjYNn78eKeh85v7Y4895oA///zzDds+/vhjB/yhhx5yd/fnn3/eAX/rrbcaxfPtb3/b27ZtGzPehQsXOuDPPvtso+033HCDA75x48ao5+3atcvNzB944IGGbVdeeaX36tUr5r3cQ78LgP/hD3+Ie4xIrpi+dL33vmN2o3dO7ztm+/Sl65t9bWCJJ/GOzZleVS1RLhQZI73xxhs8/fTTvPfee3z00Uds2bKFAQMGxD2nb9++jUbSf+ELXwBCU7N079495nlf//rXG/67W7duFBUVsXnzZgCWLl3KYYcdxkknndTonBNPPJHXX3895jUXLVoEwODBgxtt79evX6PPtbW1PPnkk8yePZsVK1awZs0a3J3PPvss3qOyd+9efv/73/P666/z4YcfsmbNGoCE54nUy3aPpvp7ZTMGJY5m6FpaQkWUJJHRImOYp59+mssvv5yxY8cyfPhwevTowWWXXdaojSOayO65RUWhXwtP0Pgefp6ZUVhY2HDOjh07aNu27QHnJGqwdnfM7ICY9uzZ0+jzD3/4Q2bOnMm4ceP46U9/So8ePejcuXPcawOcd955fPzxx4wZM4YTTjiB4uJivvjFLyY8TwRSMAvtzk3w/A/gwknQ8fAmxzGkX3lGE0UkJY5mGDWwV6NfIkjvMP9wBQWh5qnq6uqGbVOnTuWUU07h9ttvb9j3wQcfcOyxx6Y9nki9evVi48aNrFy5kl69Qn8f7t7QiB/vPHfnzTff5JxzzmnYPmfOnEbHTZ06lRtuuIGrr74agJUrVx6QIAsKChr9/Wzbto1XX32Vhx9+mEsuuQSAV155pcnPKPmn2bPQzrsL1i6Geb+C79x3wO7w0kxp+2LcYUdlVVZKFfHkSuN4i5TpYf7h6quQpk6dynvvvcfWrVvp2rUr7777LrNmzWLJkiVceeWV7Nq1K+2xRHP55Zdz6KGHcsEFF/Dqq6/y1ltvMXz4cNavXx/3vEGDBtG7d2+uvvpqpk2bxtKlS7nttttYuHBho+O6du3KzJkzWbx4MXPmzOGHP/whhYWN5+o55phj+OSTT5g9ezbvvPMOHTt2pGPHjjz99NMsXbqUF198kZ///Ocpf3ZpvZpVPb1zEyx/Crw29OfOzY12R3a22ba7iu2VVVkbqxGPEkczZXKYf7ju3bszduxYpkyZwumnn866desYO3YsAwYM4JJLLmHIkCF885vfPKCNIVNKS0t5+eWXOfjgg/nud7/L+eefT8+ePbnqqqsOeMGHKyoqYtasWXz5y1/m0ksv5ZxzzmHXrl2MGTOm0XGPP/44xcXFnHnmmfzHf/wHY8eObahiq/fv//7vnH766QwdOpSRI0dSVFTE1KlT2bBhA6eeeir//d//zfjx49Px+NJKJerRFLd7/ry7QkkDQn/O+1Wja0QrzYSrrKphzMz3m/cAKWKJ6rFbov79+/uSJUti7l+xYgXHHXdcBiOSeldddRVz5sxh7dq1iQ/OEfp9kXqRbRwQqp4eP7QvQMx9Q3oWwgPHQ3VYW11RO7jxvYa2jh6jXyKZt/H9F5+Qti+oZvaOu/dPdJzaOCRjdu7cyUsvvcTZZ5+d7VBEmiRej6bTJsyJ3f5R8eL+0kadmpoaZtw/kp/suoKupSWUti9m2+7Ea7+ka1W/IJQ4JC327NnDt771LX784x/Tu3dv1q5dyy9+8Qt27drFz372s2yHJ9JksXo0xW3/WP8W1DTuvFHoVfSuWtHQhlFcYBQXGlU18csdG7ZXZr1LsBKHpEVxcTFHH300N998M1u2bKFTp06cdtpp/OY3v6Fv377ZDk8k5eJ2z7++8WQXp02Yc8CxVbVOaUkxHdoWsWF7JWZQGyWHlLYvbl6X4BRQ4pC0KCws5Kmnnsp2GCIZE6R7fqzSyY7KKpb/57eA2O0p7jSvS3AKqFeViEgKBOmen8x8U7Gut6MyejtIJmesUIlDRCRFkh3RnWzpJNr17n5lZdZnrFCJQ0Qkw5ozeHjUwF6UFDceC5WpGSvqqcQhIpIFTZ1vSpMciohIYNme5FBVVSIiEohKHCKS91I9oC6tA/RSNDV7c6jEISlzzDHHcP3112c7DJFAUr0EdNqXlA6fmj1LlDhasA0bNrBjx46UXvOzzz5jy5YtKb1mLt1PJFK8NTZy4XqNJJiaPVOUOFqohQsXctRRR7FixYqUXXP9+vV06dLlgEWT0iXT9xOJJtVLQKd1SekEU7NnihJHKuzcBI8Nzmj237dvX8KlXYOqrq6mpib2egCplun7iUSTzCjubF6vQX1po36yxJp9WSt1KHGkQobrHK+66irOPPNMAL7xjW9gZg1Lsu7YsYMRI0bQuXNnDjnkEAYOHMj77+9f/GX9+vVccskllJWVUVJSwte+9jV27tzJmDFj6NGjBwCXXHIJZsakSZNixjBnzhxOPvlk2rVrx9FHH81DDz10wDEvv/wyZ555JocddhiHHHII3/ve9xpWAIx3vylTpnDKKadw8MEHU1ZWxhVXXMH27dub+9cmElWqB9SlbYBeeGmjXpZKHUoczZWFOse77rqLJ598EoDnnnuOVatWcfLJJ1NdXd2QKJ577jlee+01SkpKOOecc/j8888BuPDCC1m9ejUzZ85k/vz5XHbZZQCMHDmyIfncf//9rFq1igsuuCDq/RcuXMjAgQM59thjmTt3LpMnT2by5Mls2LCh0XHTp0/n/PPPZ/bs2TzzzDO8/fbb/PjHP054vxdeeIFrr72WuXPn8tvf/pYXXnjhgBUARVIl6CjuuKv8NeF6SYsyNTs1+0LbM83dW93PiSee6PH87W9/i7s/kFk3uf9XZ/f/7BT6c9ZNqbt2HHPnznXAFy1a1LBtypQpXlJS4p999lnDtk8//dQBf/rpp93dvWPHjn7NNddEveaaNWsaHRvLt771Le/Tp4/X1NQ0bNu8ebMXFhb6ddddF/O8W2+91Q855JDA97vooou8X79+cY9Jp5T+vkiLNn3peu99x2zvftuLDT+975jt05euz3ZoKQEs8STesRrH0Ryx6hwH3JaV/tULFixgz549HHnkkQfs+/vf/w7AHXfcwejRo/nggw8YOXIk559//gFrdSeyaNEiRowYQUHB/gJrly5dKC9v/I1q9erVPPzww7z99tv8/e9/p6KiIqk2jWXLljFp0iSWLl3KRx99xMaNG+nWrVugGEXiaeo4i3g9prK9Kl8mqaqqOXKozhGgTZs2HHbYYSxfvrzRz4oVK7juuusAuPXWW1mxYgUnnHACw4cP56STTmLbtm2B7uPutG3b9oDte/bsX0959erVnHDCCWzcuJHbb7+dP//5z0lVN82fP5+TTjqJwsJCfvnLX/L2229z9dVXB4pPJJ7mjLNIa4+pFkQljubIYp1j/bf96urqhm1f+cpX+PTTTykoKOBLX/pSzHN79erFxIkT+dGPfsRxxx3HM888w/XXXx/1mrHOr2+fqPf+++83Go8xa9Ysdu/ezSOPPEJxcXHDMYme4bnnnuOII47gvvvua9j217/+NW48IkE0p9QQd5W/PKLE0RwRy0FmUvfu3QGYOnUqnTp14sgjj+TSSy9l3LhxfO9732PChAn06NGDDz74gCeeeIJZs2YBcOWVV3L55ZfTtWtXXnvtNQB69+4NwBFHHEHbtm2ZPn06xx9/PKWlpRx99NEH3Pumm27isssu40c/+hHDhw9n48aN3HnnnXTq1KnhmK5du+Lu3H///QwcOJA//vGPvP76642uE+1+Xbt2ZdOmTUyePJmvfvWrTJ48mVWrVnHQQQel5e9R8k9zSg1BVvlrzVRV1UJ1796dsWPHMmXKFE4//XTWrVtHu3btePXVV/niF7/IZZddxqmnnsq4ceM4++yzG8779NNPueiii+jfvz+PPfYYkydP5owzzgBCVV3/+7//y4IFCzjllFNYtmxZ1Htfeuml3HPPPcycOZNTTz2VO++8k3vvvZeysrKGY4YNG8aNN97IhAkTGuKr71FVL9r9Ro4cyfe//31GjhzJOeecQ8eOHbnwwgtT/xcoWZWoZ1K67nfM6JeINfopmVJD2npMtTDmKR5Elgv69+/vS5Ysibl/xYoVHHfccRmMSFoy/b6kVqy1tNP1Ao52v0glxYVccGI5cz/YmrU1LnKBmb3j7v0THacSh4hkVFrnckryfuEKzbjgxHJeeKcifRMTtjJKHCKSUZnumZTourXuzP1ga0aTWUunxnERyahYPZNK2xdz2oQ5Ka8qinW/8P3qZhuMShwiklHR5nIqLjT+uac6LVVF0e5Xr75HVNomJmyllDhEJKOi9Uzq0KaIqtrGHXVSVVUUfj8ItWlA4x5RaZuYsJXK26oqd8fqfoFEYmmNvQ5zwZB+5Y2qoXqMfinqcamqKoq8X7T9QPqWe21l8jJxFBcXU1lZSfv27bMdiuS4ysrKhpHvkj65MCI7UXKR/fKyqqpLly5UVFSwe/dufaOUqNyd3bt3U1FRQZcuXbIdTquXqqqiTA8szFd5WeKonxpjw4YNVFVVZTkayVXFxcUcfvjhjaZSkfRIRVVR5EC/+gb28OtLauTEyHEzawf8D3AG0Ba4391/HbZ/EnAOUF+WPcvd18a6XqKR4yLS+pw2YU7U6q7y0hIWjD4rCxG1PMmOHM+VEkcH4BXgOuAw4H0ze97d14Udc6m7v5GN4EQk/ZJdIyPyuDN7lzH3g60xx2poLEbq5UTicPdPgRfqPv7DzNYBpcC62GeJSGuRbDVTtOOeXByz8gHQWIx0yLnGcTPrA7QDwhdhqAIeN7P3zeyWGOeNMLMlZrZk69atmQhVRFIk1vxVY2e9n/C4eDQWIz1yKnGYWWfgCeAHHtb44u7Xunt3YBBwrZmdHXmuu//O3fu7e//w6b1FJPfFqk7atruqUc+oINVO+TrleSbkTOIws0OAF4Gfufvb0Y6pa/N4EeiTydhEJL3iVSeFjx6PdlwZ23imzX9RxvaGbfUN4koa6ZETicPMOgGzgHHuPjvK/p51fx5GqNQRNbGISG6LNc4iXnVSeCkj2niPkUXTOclWckPRtEbHSfrkROIARgL9gPvNbHXdzy1m9pO6/f9jZh8DC4HfuPuCbAUqIk1T37AdbSLDIf3KKS2JPkI/vJQROe9UGdsYVjiPAnOGFb5JGdspLSlWSSPNcqVX1ThgXJz9385gOCKSBvEWcBrSr5wx530lqfW866cGmbGsgl3TRmJ1i8EWUMuNxdM56LwH0v8weS5XShwi0solWvMi6Hrebfds5YKCebS16tBnqw593qNelemWEyUOEcmcZAfapVoyExkGmWiw8k/jG0ob9QqoZfefxsM3nmlesBKXShwieSReO0O6pXrNi15VKxpKG/XaWjW9q1Y0OUZJjkocInkkUTtDOqV6zYsR7e+PPTdVsyKVRFTiEMkjrWltba3alz0qcYjkkWwumJTqac+1al/2KHGI5JFRA3sl1eW1OWI1vqejmkyr9mWHEodIHkn3t/R4pYrWVE2W75Q4RPJMOr+lxytV5MK64pIaahwXkZSJV6pQY3brocQhIikTq/TQtbQk8MhwyV2qqhKRlI0mT9T4rsbs1kGJQyTPpbKbrLrI5gclDpE8l+pusipVtH5q4xDJc+omK0EpcYjks52bmFYyrtGyq/XUTVZiUeIQyWfz7uIEX8FNbWY02qxushKPEodIvtq5CZY/heFcXDSPvgfvCdZNducmeGww7NyckXAld6hxXCRfzbsLvBaAQnNm9V0A37kv2PlrF8O8XwU7T1o8lThEctyMZRWcNmEOPUa/xGkT5qRm0aW60gY1+0Kfa/aFPidbeqg/32uDnSetghKHSA5L24p9YaWNBl4bKj0EPT/IedIqKHGI5LB4YyyaZf1b+0sb9Wr2hbbXiVnSaW5pRVo8tXGI5LC0jbG4fn7c3XFHk1fcG7u0oraOvKDEIZLDUjEVeVPmoYo7mrw0cWlFWjclDpEc1twV+5o6D1Xcks7o+KUVaf3UxiGSw8KnIu/CNqaXjOO+bx+R9FxQTW0jiTc9uogSh0iOG9KvnAWjz+Ktby6lHx8w+NPJSZ/b1DYSLbok8ShxiLQETRw30dSSgxZdknjUxiHSEkQbN5FED6ZobSQAu/dVM2NZRdxEoOnRJRaVOERyXTPGTdSXHEpLihtt37a7KjUDCSUvKXGI5LpmjvIe0q+cDm0PrFxIyUBCyUtKHCK5LolR3olosSZJJbVxiOS6BKO8k5GKgYQi9VTiEMkD6l4rqaQSh0geqO8dFXTqEZFolDhEWqigc1Cpe62kSqCqKjMbmK5ARISkl2MNuk5HWhaDkrwVtI1jtpmtMbOfm9lRaYlIJJ+FL8caR5A5qNK2GJTkraCJ43ngcGAssMbMXjKzIWZWmOA8EUlk5yZqlj4JXkvl20/w3fEvxHy5B+lem7bFoCRvBUoc7n4RcARwLbAAGAS8AKw3swlm9sWmBGFm7czsd2b2oZl9YmY3RezvY2bv1u2baGbqDSatzkcv/CfVNaEXfAG1DNs1JWbJIMgcVBrDIakW+AXs7p+7+yPufgbQHbgd+BS4FfjAzN4ws0vNrG2Ay3YAXgF6AScCo83s6LD9DwKjgWOBrwLnBY1bJKft3ET5x9Noa9UAtLVqhhW+yUFVn0YtGQTpXhueTMrYxjNt/osytmsMhzRZs765u/t6d5/g7n2ArwFTgdOBycAGM/u1mSXsxuHun7r7Cx7yD2AdUApgZmVAD3ef7e41wFOESjoiOaVZDdBRphUpoJYbiqZFLRkEmb02PMmMLJrOSbaSm9rM0BgOabJmd8c1s27A9+t+jgcMWE6oLeRG4HozG+3uDyR5vT5AO+CvdZuOAtaGHbIeOLe5cYukUlNX2muw/q2G0ka9tlbNiQWrYpYMku1eW3/Moy8vYtieeRSYc3HhPAp7qje+NE2TfnPM7AjgIkLJ4mRCyWI78BDwe3dfbmZFwFDgTuA+M/unuz+S4LqdgSeAH7i7121uA4R/FasFaqKcOwIYAdCtW7emPJZIk8VdozuZxHH9/AOSD4Sqn8anoGQwpF85QyoWwjKDGig0T3pqdpFIQcdxjDCzOYSqkn4NnAK8CVwOdHX3H7v7cgB3r3b3Zwkllo+AnyS49iHAi8DP3P3tsF0bgfD/846qu38j7v47d+/v7v3LysqCPJZIszWrAbpu7MaQnkXpWzypGVOzi0QKWuL4Td2fG4HHgUfdfXW8E9x9l5ktAb4T6xgz6wTMAsa5++yI89ea2S4zOwP4M6EkdXvAuEXSqlmTCIaN3RjynfvSM7o73tTsKnVIQEEbx2cBQ4Bu7v6zREkjzGvAHXH2jwT6Afeb2eq6n1vMrL6UciUwEfgYeNPdmz9dqEgKNXkSwSYuCRtYCqZmF6ln+5sSWo/+/fv7kiVLsh2G5Jmgc0cB8OLNsOyJ0Eu8sA30u1wlAMkaM3vH3fsnPE6JQyS+JiWEZOzcBA8cD9V79m8ragc3vgcdD2/+9UUCSjZxqD+eSBzN7mYbdp0Dkk/FvQe2O1TvhT+NgfMfStUjiKScpu4Qiaaup9MjLy9u9jxPsSYZ3P7hggPbHXD48OXmxy+SRkocItHU9XQatmtK1N1B5nmKNcbj3H3jYcyO0M8tK0PVVABVu9VNVnKaEodIpLCeThcVvUkZ2w84JMg8T0mN8QjvLlvfTVYkRylxiEQKe4kXm3NTmxmNdgddq/vgkuL42zU4T1oYJQ6RcBEv8UKv4uKiefQ9eE+TR3ObJdgeb3CeSA5SryqRcFFe4oXmzOq7oMnjK7bvroq/XYPzpIVR4hAJF+sl/snCJl8y4XQk12siBGlZVFUlEu76+ft7Oo3ZAf2vBiuA7qc2+ZJNno5EJEepxCE5KW2jtYOInEdqwG1NGtFdH3fWn0ckRZQ4JOekarR2s0XrItvEdo5kF10SaQlUVSU5J96iSBmjLrIiMSlxSM5p1qJIqaIusiIxqapKckJ4m0aBGTVRZm0OMlq72RJ0kc2JNhiRLFHikKyLbNOIljSC9EJKyUs9ThfZnGmDEckSJQ7JumhtGgCFZtS6x335RyaJM3uX8cI7FWl9qcdrg1HikHygxCFZF6vtotadNRPOjXletG/+Ty1eS2R5JdUv9ZxogxHJIjWOS9bFarsoMKPH6Jc4bcIcZiyrOGB/tG/+sdazTOVLPVa8GW2DEckiJQ7JumgjqyHU1hG+8FFk8giSDFL5UtdIcMl3ShySdUP6lTN+aF/KS0swQm0bkaKN40g2GZQUF3LHgEPhscEpGYcRGW9TZswVacnMo/Rgaen69+/vS5YsyXYY0kQ9Rr8UtcrJoFGbR2QbRzTl4et7v/MYnPiDJo/+FmntzOwdd++f6DiVOCTnJNuGUP/NPxYDFow+iyE9CxvPOaXR3yLNosQhOSdIG8KQfuWUJ0o0WpZVJKWUOCTnBG1DiJtoNOeUSMppHIfkpCCzycadtvzFm2PPOaW2DpEmUeKQViFmotGyrCIpp8QhrZuWZRVJObVxiIhIIEockl07N6VsYJ6IZIYShzTbjGUVnDZhTtx5pWKadxesXawusiItiNo48kS6Fh5KtDZF+H0PLinGDLbvrqJraQl3DDiUweED8wbcBh0PDxbAzk3w/A/gwknBzxWRJlGJIw/Uv9wrtlfGnTSwKeKtTRF53+2VVWzbXdUQw/bZ46ipqTu3qQPzVGIRyTgljjwQ7+XeXPHWpoi1QBNAGds4396g0KtCG5oyMK9+cJ+mEhHJKCWOPJDOhYfizSsV7/oji6ZjkVMZBi11aCoRkaxQ4sgD6Vx4KN50H/Gu/7WCVbS16sYbgwzM01QiIlmjxJEH0rnwULx5pWIt0ARw7r7xHFfzDLMHzoPup8ItH8KYHY0G7MXkwzmKAAAOjElEQVTtrRVe2qinUodIRqhXVR6IO5dTiq4f7VqR943sVTVqYC8GV9yLf7KIafeP5Ce7rmjYDsTtraWpRESyJ6cWcjKzEuBod/+wOdfRQk4txM5N1Pz6qxTW7qXS2/Ave+9nK6WUFBfSrriAbburDjilvLSEBaPPykKwIq1fi1rIycw6mdkMYDNwa5T9k8yswsxW1/10y3yUkqykBwTOu4ua2lCJooBabiiaBoR6fEVLGpCaBn0RaZ6cSBxALTARuDnOMZe6e8+6n7UZiksCSnrMSF3jdhtCDeRtrZphhW9Sxva4109Fg76INE9OJA53/6e7vw5UJzxYclrSY0aiNG6HlzpKS4rT1qAvIs3TUhrHq4DHzeyfwKPufm+2A2qNUjEtSdJjRqI0bre1ak4sWEVJcSFjzvsKkL4GfRFpuhaRONz9WgAzOxp4zczedfc/hR9jZiOAEQDduqkJJFKipJBozqlkdS0toSJK8jigiimi2214bOPDYlOiEMk9LSJx1HP3dWb2ItAH+FPEvt8Bv4NQr6oshJezkkkK8aqYgry8Rw3s1ehekLiKKcgysSKSfS0icZhZT3dfbWaHAYOA67IdUyqka8baSLGSwi3PvstNzyyPWUqA4L2Y0j1mRESyLycSh5l1BJYBHYF2ZnYGMAr4grvfA/yPmX0Z2AtMdPcFWQs2RVJVNZSMWC//mroxPBXbKzGInDkKaFovplgliEwlShFJr5xIHO6+E+gZZ/+3MxhORqSqaigZ8UoU9RwOSB6p7MWUyUQpIumVE91x81E6Z6yNFG/OqHAOUeecSoV0Tu0uIpmVEyWOfJR076MUiGx3KDBrqKYKl87pPDKZKEUkvVTiyJJ0zlgbzZB+5SwYfRZrJpzLvRcdn/HBdemc2l1EMkuJI0viTUfeGu+d6UQpIumTU7Pjpopmx20sV3oz5UocIhJdsrPjqo2jlap/SUd2tQ3vzQSZHW+hgX4irYMSR2uxcxM8/wO4cBIzVlc36voaWaasrKphzMz32Vtdq+6xIhKY2jhai3l3wdrFMO9XUbu+RtpeWaXusSLSJEocrUHd2hZ4LSx/in3bNzb5UuoeKyKJKHHkqKRX0YPGa1t4LaM7zIx77ZLiQg5pXxx1n7rHikgiauPIQfGm54DGDdp3DDiUwcuf2r+2Rc0+hhTM5f7i81lX1bHhnPoG8vK6RnAg8Cy2IiKgxJFb6hq4H9l0DZVV7RrtitWgvX32OGoKawgfIVFozuNfmMvlmy5O2GNK3WNFJCgljlxS18A9rKo9f2H4Abu3V1YdsO2rfEihR2yv2cexe95POH2IuseKSFMoceSKsAbui4reZGL1ULZSmvC0c/eNP2BbSXEh40/uy5B0xCkieU+N4ykUqEE7UlgDd7E5N7WZ0Wh3vAbtSMl0q21WrCKS15Q4UqS+QbtieyXO/gbtpF7I9aWNugbuQq/i4qJ59D14T6O5pP7zu19Janp0iN+ttlmxikjeU1VVijRrYabw7rR1Cs2Z1XcBfOe+qPeqb9Detbc6attHvG61mVxESkRaHyWOFGnWehPr39rfnbZezT62f7iAcyfMOaDXU/jLPbLrLiTuVqu1MUSkOZQ4UqRZCzNdP/+ATfsTQuiaseaSilykKZlutZlcREpEWh8ljhQZNbBXSgfUBalOCtqtNtWxikh+UeJIkaZ8848nndVJqY5VRPKLFnJKg1QsWHTahDlRq5PSuS64iOS3ZBdyUnfcAJIZ+5Cqrq5aalVEcpUSR5KSTQjR2iYOqvoHR04fynfHv5B0AsnmmuQiIvGojSNJyTZWR2uDGFk0nZNsJcN2TeGn0zoA8VfZi6zq+vXFJyhhiEjOUIkjSck2Vkd2aS1jG8MK51FgzrDCNzmo6tO404FoVLeI5DoljiTFGuMQuf3M3mWNPo8smo7VrfpdQC03FE2L2zMqXslGRCQXKHHEEd4YvmtvNcWF1mh/SXEhdww4FB4bDDs3AzD3g60N++tLG22tGoC2Vs2wwjfpc/CemPfUqG4RyXVKHDFEVhltr6wCh0PaFzdqrB786WRYuxjm/Qpo/IIPL23UK6CWB458NeZ9kynZaGZbEckmJY4YolUZVdU67dsUsWbCuSwYfRZDehY2rKHB8qdg5+ZGL/ivFaxqKG3Ua2vVHLvn/Zj3TdQNV20gIpJt6lUVQ1JVRuGz2notzPsVowbe0jCdR/0iSyXFhUl3pU00qlsz24pItilxxJBwIsCINTSo2QfLn2LIgNtgaN9mjRyPN/eU2kBEJNuUOGKINhEgwK691cxYVsGQinsPWEOjvtQx5Dv3pe3bv2a2FZFsUxtHDPUjtyOXa91eWcVPp/2F7R8uiLqGBuvfinvd5jZsayoSEck2lTjCRJucsH2bIrbtbrzCXn37xYIxwSYbjFx0KdYaG/FoZlsRyTbNjlsn1kp6kVVV9QxYM+HcQPfQjLcikss0O25AsXorFZpFPb4pbQpq2BaR1kCJo06sl3eNe8raFJKdtkREJJcpcdSJ9fKuHyGeiunN1bAtIq1BTjWOm1kJcLS7f5jpe8dbhzvomt6xqGFbRFqDnEgcZtYJmAycBTwLXBOxvw/wFFAKzARudI8cRNE8TXmpN2WJ2FQlIRGRbMmJxAHUAhOBF4FToux/EBgNvArMAc4DZqQ6iCAv9VR0rRURaYlyoo3D3f/p7q8D1ZH7zKwM6OHus929hlDJY1CmY4ykdTNEJF/lROJI4Chgbdjn9cCRkQeZ2QgzW2JmS7Zu3Rq5O+XUtVZE8lVLSBxtCFVl1asFDhiV5+6/c/f+7t6/rKwscnfKqWutiOSrlpA4NgLhjQZHAeuyFEsDda0VkXyV84nD3dcCu8zsDDMrBC4HnstyWA2TIKZifIeISEuSE72qzKwjsAzoCLQzszOAUcAX3P0e4ErgcULdcSe5+/xsxRpOXWtFJB/lROJw951Azzj7lwJ9MxeRiIjEkvNVVSIikluUOEREJBAlDhERCUSJQ0REAmmVKwCa2Vbgkyac2hn4R4rDaQn03PlFz51fgjx3d3dPOIK6VSaOpjKzJcksm9ja6Lnzi547v6TjuVVVJSIigShxiIhIIEocjf0u2wFkiZ47v+i580vKn1ttHCIiEohKHCIiEogSh4hIK2JmJWb2pXTeIy8Th5ldZGZrzGy1mQ2P2NfHzN41s0/MbKKZtZq/owTPfaOZrah77ifMLCcmwEyFeM8ddswjZrY607GlU6LnNrMxZrbOzD42s1OzEWM6JPg9P9vM/lK3/9G6pRpaBTPrZGYzgM3ArVH2p+7d5u559UNo6vZ1hBaHOgLYBJSF7X8TGAwUAvOAIdmOOUPPPZzQaotFwKvApdmOORPPXXfMmcBLwOpsx5vhf+8XgRLAgHbZjjlDz70G6FP3//d8YHC2Y07hsx8E/CtwDfD7KPtT9m5rNd+mAxgIzHP3CnffBMwh9JeNmZUBPdx9trvXAE8Bg7IXakrFfG4Ad3/U3fe5ezXwHnBoluJMtbjPbWbtgHHAz7MUX7rEfW7gJuA/3L3SQ/ZkJcrUS/Tce8P+uy2wJZPBpZO7/9PdXweqI/el+t2Wj4njaBpPR7IeOLLuv48C1sbY19LFe+4GZtYeOBeYlaG40i3Rc98JPAh8lsmgMiDmc5tZMaFv41eb2Uozm25mh2UhxnRI9O99GfAM8H/Ak+7+TgZjy6aUvtvyMXG0AWrDPtcCNUnsa+kSPltdnedkYKK7f5y50NIq5nObWV/geHd/KhuBpVm8f+/OwCGEvo33JvRCuT2j0aVPot/za4GHgJHABWbWLYOxZVNK3235mDg2Eqr/rHcUoTrRRPtaurjPZmYG/B74m7s/mOHY0inec18J9DSz5cAfgaPN7JkMx5cu8Z77H8A/3f01D1V+/wHoleH40iXmc5vZl4ET3f1/3X0hMAP4UeZDzIqUvtvyMXG8Cgw0sy5mdgRwat023H0tsMvMzqjrbXE58Fz2Qk2pmM9d5yFgk7vfmZXo0ifev/dP3L2Xu58AfBtY5+4XZzHWVIr33FXA/5lZfR33d4C3sxNmysX7Pd8LdDOzw+tK1/2AbVmKM6NS/W7Ly5HjZnYV+xtDf1L35xfc/R4z+xrwOFAKTHL3VtNoGuu5gcWEelx8FHb4z9396cxFlz7x/r3DjjkG+JO798xocGmU4Pf8WOAJ4HBCSeMad9+V+ShTL8Fz/4RQNdVe9j/37sxHmXpm1hFYRqhnWTtgKzCKNLzb8jJxiIhI0+VjVZWIiDSDEoeIiASixCEiIoEocYiISCBKHCIiEogSh4iIBKLEISIigShxiKSJmR1hZtvNzM1sWIxjOpvZp2ZWa2bfzHSMIk2hxCGSJnXTev+s7uPddVO4R7qb0BT2v3X3+RkLTqQZNHJcJI3q5kRaBHwduNPdfxG27zTgz4QmoPuyu+/ITpQiwShxiKSZmfUjNC/SHqCXu1fULc27FOgLDHX36dmMUSQIVVWJpJm7LwMmAh2AX9VtvpFQ0piupCEtjUocIhlQN3Pp3witiXAOMK1u13HuviFrgYk0gUocIhng7jsJlTIMmAl0Am5T0pCWSCUOkQypW2VxDdAd+Bzo4u57sxuVSHAqcYhkznWEksZmQiWOUdkNR6RpVOIQyYC6FQb/AlQD/wIsBIqA4939w+xFJhKcShwiaVZXRfUIcBDwU3f/C/BfhJb3/G02YxNpCpU4RNLMzH4IPAj8H3Cqu9eaWTHwHtCb0LrXj2QzRpEglDhE0iisiqoE6O/uy8P2/SvwJ2AboW65m7MRo0hQqqoSSZO6KqpHCVVRTQxPGgDu/jrwHHAI8EDmIxRpGpU4RNLEzP4d+H/ABqB33ViOyGOOAj4gNKr8XHf/Y2ajFAlOiUNERAJRVZWIiASixCEiIoEocYiISCBKHCIiEogSh4iIBKLEISIigShxiIhIIEocIiISiBKHiIgEosQhIiKBKHGIiEgg/x/nbr8i5XlarAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x_train, y_train, 'o', label = 'taining data')\n",
    "plt.plot(x_val, y_val, '^', label = 'test data')\n",
    "plt.xlabel('X', fontsize = 20)\n",
    "plt.ylabel('y', fontsize = 20)\n",
    "plt.legend(fontsize = 15);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Linear Regression in Numpy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For training a model, there are two initialization steps:\n",
    "1. Random initialization of parameters/weights (we have only two, a and b) — lines 3 and 4;\n",
    "2. Initialization of hyper-parameters (in our case, only learning rate and number of epochs) — lines 9 and 11;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:36:04.454411Z",
     "start_time": "2019-06-19T08:36:04.445588Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.49671415] [-0.1382643]\n"
     ]
    }
   ],
   "source": [
    "# Initializes parameters \"a\" and \"b\" randomly\n",
    "np.random.seed(42)\n",
    "a = np.random.randn(1)\n",
    "b = np.random.randn(1)\n",
    "print(a, b)\n",
    "\n",
    "# Sets learning rate\n",
    "lr = 1e-1\n",
    "# Defines number of epochs\n",
    "n_epochs = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For each epoch, there are four training steps:\n",
    "1. Compute model’s predictions — this is the forward pass;\n",
    "2. Compute the loss, using predictions and and labels and the appropriate loss function for the task at hand;\n",
    "3. Compute the gradients for every parameter;\n",
    "4. Update the parameters;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:36:07.268256Z",
     "start_time": "2019-06-19T08:36:07.220623Z"
    },
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.02354094] [1.96896411]\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(n_epochs):\n",
    "    # Computes our model's predicted output\n",
    "    yhat = a + b * x_train\n",
    "    # How wrong is our model? That's the error! \n",
    "    error = (y_train - yhat)\n",
    "    # It is a regression, so it computes mean squared error (MSE)\n",
    "    loss = (error ** 2).mean()\n",
    "    # Computes gradients for both \"a\" and \"b\" parameters\n",
    "    a_grad = -2 * error.mean()\n",
    "    b_grad = -2 * (x_train * error).mean()\n",
    "    # Updates parameters using gradients and the learning rate\n",
    "    a = a - lr * a_grad\n",
    "    b = b - lr * b_grad\n",
    "    \n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:36:10.442277Z",
     "start_time": "2019-06-19T08:36:09.763897Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.02354075] [1.96896447]\n"
     ]
    }
   ],
   "source": [
    "# Sanity Check: do we get the same results as our gradient descent?\n",
    "from sklearn.linear_model import LinearRegression\n",
    "linr = LinearRegression()\n",
    "linr.fit(x_train, y_train)\n",
    "print(linr.intercept_, linr.coef_[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "if we don’t use batch gradient descent (our example does)\n",
    "- we have to write an inner loop to perform the four training steps for \n",
    "    - either each individual point (stochastic) or \n",
    "    - n points (mini-batch)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Tensor\n",
    "- A scalar (a single number) has zero dimensions, \n",
    "- a vector has one dimension, \n",
    "- a matrix has two dimensions and \n",
    "- a tensor has three or more dimensions. \n",
    "\n",
    "<img src= 'img/neuralnetwork/tensor.jpeg' width= \"800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loading data: turning Numpy arrays into PyTorch tensors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:36:15.271099Z",
     "start_time": "2019-06-19T08:36:15.259046Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'> <class 'torch.Tensor'> torch.FloatTensor\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import torch.nn as nn\n",
    "\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "# Our data was in Numpy arrays, but we need to transform them into PyTorch's Tensors\n",
    "# and then we send them to the chosen device\n",
    "x_train_tensor = torch.from_numpy(x_train).float().to(device)\n",
    "y_train_tensor = torch.from_numpy(y_train).float().to(device)\n",
    "\n",
    "# Here we can see the difference - notice that .type() is more useful\n",
    "# since it also tells us WHERE the tensor is (device)\n",
    "print(type(x_train), type(x_train_tensor), x_train_tensor.type())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:36:15.908827Z",
     "start_time": "2019-06-19T08:36:15.898651Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.3367], requires_grad=True) tensor([0.1288], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "# We can specify the device at the moment of creation - RECOMMENDED!\n",
    "torch.manual_seed(42)\n",
    "a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:23:10.742904Z",
     "start_time": "2019-06-19T08:23:09.736269Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-3.1125])\n",
      "tensor([-1.8156])\n",
      "tensor([-2.3184])\n",
      "tensor([-1.4064])\n",
      "tensor([-1.7219])\n",
      "tensor([-1.0982])\n",
      "tensor([-1.2737])\n",
      "tensor([-0.8659])\n",
      "tensor([-0.9372])\n",
      "tensor([-0.6906])\n",
      "tensor([-0.6845])\n",
      "tensor([-0.5583])\n",
      "tensor([-0.4948])\n",
      "tensor([-0.4582])\n",
      "tensor([-0.3526])\n",
      "tensor([-0.3824])\n",
      "tensor([-0.2459])\n",
      "tensor([-0.3248])\n",
      "tensor([-0.1660])\n",
      "tensor([-0.2810])\n",
      "tensor([-0.1063])\n",
      "tensor([-0.2475])\n",
      "tensor([-0.0616])\n",
      "tensor([-0.2218])\n",
      "tensor([-0.0283])\n",
      "tensor([-0.2019])\n",
      "tensor([-0.0036])\n",
      "tensor([-0.1864])\n",
      "tensor([0.0147])\n",
      "tensor([-0.1743])\n",
      "tensor([0.0283])\n",
      "tensor([-0.1646])\n",
      "tensor([0.0382])\n",
      "tensor([-0.1568])\n",
      "tensor([0.0453])\n",
      "tensor([-0.1505])\n",
      "tensor([0.0505])\n",
      "tensor([-0.1452])\n",
      "tensor([0.0541])\n",
      "tensor([-0.1408])\n",
      "tensor([0.0566])\n",
      "tensor([-0.1370])\n",
      "tensor([0.0582])\n",
      "tensor([-0.1337])\n",
      "tensor([0.0592])\n",
      "tensor([-0.1307])\n",
      "tensor([0.0597])\n",
      "tensor([-0.1280])\n",
      "tensor([0.0599])\n",
      "tensor([-0.1255])\n",
      "tensor([0.0598])\n",
      "tensor([-0.1232])\n",
      "tensor([0.0594])\n",
      "tensor([-0.1211])\n",
      "tensor([0.0590])\n",
      "tensor([-0.1190])\n",
      "tensor([0.0584])\n",
      "tensor([-0.1170])\n",
      "tensor([0.0578])\n",
      "tensor([-0.1151])\n",
      "tensor([0.0571])\n",
      "tensor([-0.1133])\n",
      "tensor([0.0564])\n",
      "tensor([-0.1115])\n",
      "tensor([0.0557])\n",
      "tensor([-0.1098])\n",
      "tensor([0.0549])\n",
      "tensor([-0.1081])\n",
      "tensor([0.0541])\n",
      "tensor([-0.1064])\n",
      "tensor([0.0534])\n",
      "tensor([-0.1048])\n",
      "tensor([0.0526])\n",
      "tensor([-0.1032])\n",
      "tensor([0.0518])\n",
      "tensor([-0.1016])\n",
      "tensor([0.0511])\n",
      "tensor([-0.1001])\n",
      "tensor([0.0503])\n",
      "tensor([-0.0985])\n",
      "tensor([0.0496])\n",
      "tensor([-0.0970])\n",
      "tensor([0.0488])\n",
      "tensor([-0.0956])\n",
      "tensor([0.0481])\n",
      "tensor([-0.0941])\n",
      "tensor([0.0474])\n",
      "tensor([-0.0927])\n",
      "tensor([0.0466])\n",
      "tensor([-0.0913])\n",
      "tensor([0.0459])\n",
      "tensor([-0.0899])\n",
      "tensor([0.0453])\n",
      "tensor([-0.0886])\n",
      "tensor([0.0446])\n",
      "tensor([-0.0872])\n",
      "tensor([0.0439])\n",
      "tensor([-0.0859])\n",
      "tensor([0.0432])\n",
      "tensor([-0.0846])\n",
      "tensor([0.0426])\n",
      "tensor([-0.0833])\n",
      "tensor([0.0419])\n",
      "tensor([-0.0821])\n",
      "tensor([0.0413])\n",
      "tensor([-0.0808])\n",
      "tensor([0.0407])\n",
      "tensor([-0.0796])\n",
      "tensor([0.0401])\n",
      "tensor([-0.0784])\n",
      "tensor([0.0395])\n",
      "tensor([-0.0772])\n",
      "tensor([0.0389])\n",
      "tensor([-0.0761])\n",
      "tensor([0.0383])\n",
      "tensor([-0.0749])\n",
      "tensor([0.0377])\n",
      "tensor([-0.0738])\n",
      "tensor([0.0371])\n",
      "tensor([-0.0727])\n",
      "tensor([0.0366])\n",
      "tensor([-0.0716])\n",
      "tensor([0.0360])\n",
      "tensor([-0.0705])\n",
      "tensor([0.0355])\n",
      "tensor([-0.0694])\n",
      "tensor([0.0349])\n",
      "tensor([-0.0684])\n",
      "tensor([0.0344])\n",
      "tensor([-0.0673])\n",
      "tensor([0.0339])\n",
      "tensor([-0.0663])\n",
      "tensor([0.0334])\n",
      "tensor([-0.0653])\n",
      "tensor([0.0329])\n",
      "tensor([-0.0643])\n",
      "tensor([0.0324])\n",
      "tensor([-0.0634])\n",
      "tensor([0.0319])\n",
      "tensor([-0.0624])\n",
      "tensor([0.0314])\n",
      "tensor([-0.0615])\n",
      "tensor([0.0309])\n",
      "tensor([-0.0605])\n",
      "tensor([0.0305])\n",
      "tensor([-0.0596])\n",
      "tensor([0.0300])\n",
      "tensor([-0.0587])\n",
      "tensor([0.0296])\n",
      "tensor([-0.0578])\n",
      "tensor([0.0291])\n",
      "tensor([-0.0570])\n",
      "tensor([0.0287])\n",
      "tensor([-0.0561])\n",
      "tensor([0.0282])\n",
      "tensor([-0.0552])\n",
      "tensor([0.0278])\n",
      "tensor([-0.0544])\n",
      "tensor([0.0274])\n",
      "tensor([-0.0536])\n",
      "tensor([0.0270])\n",
      "tensor([-0.0528])\n",
      "tensor([0.0266])\n",
      "tensor([-0.0520])\n",
      "tensor([0.0262])\n",
      "tensor([-0.0512])\n",
      "tensor([0.0258])\n",
      "tensor([-0.0504])\n",
      "tensor([0.0254])\n",
      "tensor([-0.0497])\n",
      "tensor([0.0250])\n",
      "tensor([-0.0489])\n",
      "tensor([0.0246])\n",
      "tensor([-0.0482])\n",
      "tensor([0.0242])\n",
      "tensor([-0.0474])\n",
      "tensor([0.0239])\n",
      "tensor([-0.0467])\n",
      "tensor([0.0235])\n",
      "tensor([-0.0460])\n",
      "tensor([0.0232])\n",
      "tensor([-0.0453])\n",
      "tensor([0.0228])\n",
      "tensor([-0.0446])\n",
      "tensor([0.0225])\n",
      "tensor([-0.0440])\n",
      "tensor([0.0221])\n",
      "tensor([-0.0433])\n",
      "tensor([0.0218])\n",
      "tensor([-0.0426])\n",
      "tensor([0.0215])\n",
      "tensor([-0.0420])\n",
      "tensor([0.0211])\n",
      "tensor([-0.0414])\n",
      "tensor([0.0208])\n",
      "tensor([-0.0407])\n",
      "tensor([0.0205])\n",
      "tensor([-0.0401])\n",
      "tensor([0.0202])\n",
      "tensor([-0.0395])\n",
      "tensor([0.0199])\n",
      "tensor([-0.0389])\n",
      "tensor([0.0196])\n",
      "tensor([-0.0383])\n",
      "tensor([0.0193])\n",
      "tensor([-0.0378])\n",
      "tensor([0.0190])\n",
      "tensor([-0.0372])\n",
      "tensor([0.0187])\n",
      "tensor([-0.0366])\n",
      "tensor([0.0184])\n",
      "tensor([-0.0361])\n",
      "tensor([0.0182])\n",
      "tensor([-0.0355])\n",
      "tensor([0.0179])\n",
      "tensor([-0.0350])\n",
      "tensor([0.0176])\n",
      "tensor([-0.0345])\n",
      "tensor([0.0173])\n",
      "tensor([-0.0339])\n",
      "tensor([0.0171])\n",
      "tensor([-0.0334])\n",
      "tensor([0.0168])\n",
      "tensor([-0.0329])\n",
      "tensor([0.0166])\n",
      "tensor([-0.0324])\n",
      "tensor([0.0163])\n",
      "tensor([-0.0319])\n",
      "tensor([0.0161])\n",
      "tensor([-0.0315])\n",
      "tensor([0.0158])\n",
      "tensor([-0.0310])\n",
      "tensor([0.0156])\n",
      "tensor([-0.0305])\n",
      "tensor([0.0154])\n",
      "tensor([-0.0301])\n",
      "tensor([0.0151])\n",
      "tensor([-0.0296])\n",
      "tensor([0.0149])\n",
      "tensor([-0.0291])\n",
      "tensor([0.0147])\n",
      "tensor([-0.0287])\n",
      "tensor([0.0145])\n",
      "tensor([-0.0283])\n",
      "tensor([0.0142])\n",
      "tensor([-0.0278])\n",
      "tensor([0.0140])\n",
      "tensor([-0.0274])\n",
      "tensor([0.0138])\n",
      "tensor([-0.0270])\n",
      "tensor([0.0136])\n",
      "tensor([-0.0266])\n",
      "tensor([0.0134])\n",
      "tensor([-0.0262])\n",
      "tensor([0.0132])\n",
      "tensor([-0.0258])\n",
      "tensor([0.0130])\n",
      "tensor([-0.0254])\n",
      "tensor([0.0128])\n",
      "tensor([-0.0250])\n",
      "tensor([0.0126])\n",
      "tensor([-0.0247])\n",
      "tensor([0.0124])\n",
      "tensor([-0.0243])\n",
      "tensor([0.0122])\n",
      "tensor([-0.0239])\n",
      "tensor([0.0120])\n",
      "tensor([-0.0236])\n",
      "tensor([0.0119])\n",
      "tensor([-0.0232])\n",
      "tensor([0.0117])\n",
      "tensor([-0.0228])\n",
      "tensor([0.0115])\n",
      "tensor([-0.0225])\n",
      "tensor([0.0113])\n",
      "tensor([-0.0222])\n",
      "tensor([0.0112])\n",
      "tensor([-0.0218])\n",
      "tensor([0.0110])\n",
      "tensor([-0.0215])\n",
      "tensor([0.0108])\n",
      "tensor([-0.0212])\n",
      "tensor([0.0107])\n",
      "tensor([-0.0209])\n",
      "tensor([0.0105])\n",
      "tensor([-0.0205])\n",
      "tensor([0.0103])\n",
      "tensor([-0.0202])\n",
      "tensor([0.0102])\n",
      "tensor([-0.0199])\n",
      "tensor([0.0100])\n",
      "tensor([-0.0196])\n",
      "tensor([0.0099])\n",
      "tensor([-0.0193])\n",
      "tensor([0.0097])\n",
      "tensor([-0.0190])\n",
      "tensor([0.0096])\n",
      "tensor([-0.0187])\n",
      "tensor([0.0094])\n",
      "tensor([-0.0185])\n",
      "tensor([0.0093])\n",
      "tensor([-0.0182])\n",
      "tensor([0.0092])\n",
      "tensor([-0.0179])\n",
      "tensor([0.0090])\n",
      "tensor([-0.0176])\n",
      "tensor([0.0089])\n",
      "tensor([-0.0174])\n",
      "tensor([0.0087])\n",
      "tensor([-0.0171])\n",
      "tensor([0.0086])\n",
      "tensor([-0.0169])\n",
      "tensor([0.0085])\n",
      "tensor([-0.0166])\n",
      "tensor([0.0084])\n",
      "tensor([-0.0163])\n",
      "tensor([0.0082])\n",
      "tensor([-0.0161])\n",
      "tensor([0.0081])\n",
      "tensor([-0.0159])\n",
      "tensor([0.0080])\n",
      "tensor([-0.0156])\n",
      "tensor([0.0079])\n",
      "tensor([-0.0154])\n",
      "tensor([0.0077])\n",
      "tensor([-0.0151])\n",
      "tensor([0.0076])\n",
      "tensor([-0.0149])\n",
      "tensor([0.0075])\n",
      "tensor([-0.0147])\n",
      "tensor([0.0074])\n",
      "tensor([-0.0145])\n",
      "tensor([0.0073])\n",
      "tensor([-0.0143])\n",
      "tensor([0.0072])\n",
      "tensor([-0.0140])\n",
      "tensor([0.0071])\n",
      "tensor([-0.0138])\n",
      "tensor([0.0070])\n",
      "tensor([-0.0136])\n",
      "tensor([0.0069])\n",
      "tensor([-0.0134])\n",
      "tensor([0.0068])\n",
      "tensor([-0.0132])\n",
      "tensor([0.0066])\n",
      "tensor([-0.0130])\n",
      "tensor([0.0065])\n",
      "tensor([-0.0128])\n",
      "tensor([0.0064])\n",
      "tensor([-0.0126])\n",
      "tensor([0.0064])\n",
      "tensor([-0.0124])\n",
      "tensor([0.0063])\n",
      "tensor([-0.0122])\n",
      "tensor([0.0062])\n",
      "tensor([-0.0121])\n",
      "tensor([0.0061])\n",
      "tensor([-0.0119])\n",
      "tensor([0.0060])\n",
      "tensor([-0.0117])\n",
      "tensor([0.0059])\n",
      "tensor([-0.0115])\n",
      "tensor([0.0058])\n",
      "tensor([-0.0113])\n",
      "tensor([0.0057])\n",
      "tensor([-0.0112])\n",
      "tensor([0.0056])\n",
      "tensor([-0.0110])\n",
      "tensor([0.0055])\n",
      "tensor([-0.0108])\n",
      "tensor([0.0055])\n",
      "tensor([-0.0107])\n",
      "tensor([0.0054])\n",
      "tensor([-0.0105])\n",
      "tensor([0.0053])\n",
      "tensor([-0.0104])\n",
      "tensor([0.0052])\n",
      "tensor([-0.0102])\n",
      "tensor([0.0051])\n",
      "tensor([-0.0100])\n",
      "tensor([0.0051])\n",
      "tensor([-0.0099])\n",
      "tensor([0.0050])\n",
      "tensor([-0.0097])\n",
      "tensor([0.0049])\n",
      "tensor([-0.0096])\n",
      "tensor([0.0048])\n",
      "tensor([-0.0094])\n",
      "tensor([0.0048])\n",
      "tensor([-0.0093])\n",
      "tensor([0.0047])\n",
      "tensor([-0.0092])\n",
      "tensor([0.0046])\n",
      "tensor([-0.0090])\n",
      "tensor([0.0045])\n",
      "tensor([-0.0089])\n",
      "tensor([0.0045])\n",
      "tensor([-0.0088])\n",
      "tensor([0.0044])\n",
      "tensor([-0.0086])\n",
      "tensor([0.0043])\n",
      "tensor([-0.0085])\n",
      "tensor([0.0043])\n",
      "tensor([-0.0084])\n",
      "tensor([0.0042])\n",
      "tensor([-0.0082])\n",
      "tensor([0.0041])\n",
      "tensor([-0.0081])\n",
      "tensor([0.0041])\n",
      "tensor([-0.0080])\n",
      "tensor([0.0040])\n",
      "tensor([-0.0079])\n",
      "tensor([0.0040])\n",
      "tensor([-0.0078])\n",
      "tensor([0.0039])\n",
      "tensor([-0.0076])\n",
      "tensor([0.0038])\n",
      "tensor([-0.0075])\n",
      "tensor([0.0038])\n",
      "tensor([-0.0074])\n",
      "tensor([0.0037])\n",
      "tensor([-0.0073])\n",
      "tensor([0.0037])\n",
      "tensor([-0.0072])\n",
      "tensor([0.0036])\n",
      "tensor([-0.0071])\n",
      "tensor([0.0036])\n",
      "tensor([-0.0070])\n",
      "tensor([0.0035])\n",
      "tensor([-0.0069])\n",
      "tensor([0.0035])\n",
      "tensor([-0.0068])\n",
      "tensor([0.0034])\n",
      "tensor([-0.0067])\n",
      "tensor([0.0034])\n",
      "tensor([-0.0066])\n",
      "tensor([0.0033])\n",
      "tensor([-0.0065])\n",
      "tensor([0.0033])\n",
      "tensor([-0.0064])\n",
      "tensor([0.0032])\n",
      "tensor([-0.0063])\n",
      "tensor([0.0032])\n",
      "tensor([-0.0062])\n",
      "tensor([0.0031])\n",
      "tensor([-0.0061])\n",
      "tensor([0.0031])\n",
      "tensor([-0.0060])\n",
      "tensor([0.0030])\n",
      "tensor([-0.0059])\n",
      "tensor([0.0030])\n",
      "tensor([-0.0058])\n",
      "tensor([0.0029])\n",
      "tensor([-0.0057])\n",
      "tensor([0.0029])\n",
      "tensor([-0.0056])\n",
      "tensor([0.0028])\n",
      "tensor([-0.0055])\n",
      "tensor([0.0028])\n",
      "tensor([-0.0055])\n",
      "tensor([0.0027])\n",
      "tensor([-0.0054])\n",
      "tensor([0.0027])\n",
      "tensor([-0.0053])\n",
      "tensor([0.0027])\n",
      "tensor([-0.0052])\n",
      "tensor([0.0026])\n",
      "tensor([-0.0051])\n",
      "tensor([0.0026])\n",
      "tensor([-0.0051])\n",
      "tensor([0.0025])\n",
      "tensor([-0.0050])\n",
      "tensor([0.0025])\n",
      "tensor([-0.0049])\n",
      "tensor([0.0025])\n",
      "tensor([-0.0048])\n",
      "tensor([0.0024])\n",
      "tensor([-0.0048])\n",
      "tensor([0.0024])\n",
      "tensor([-0.0047])\n",
      "tensor([0.0024])\n",
      "tensor([-0.0046])\n",
      "tensor([0.0023])\n",
      "tensor([-0.0046])\n",
      "tensor([0.0023])\n",
      "tensor([-0.0045])\n",
      "tensor([0.0023])\n",
      "tensor([-0.0044])\n",
      "tensor([0.0022])\n",
      "tensor([-0.0043])\n",
      "tensor([0.0022])\n",
      "tensor([-0.0043])\n",
      "tensor([0.0022])\n",
      "tensor([-0.0042])\n",
      "tensor([0.0021])\n",
      "tensor([-0.0042])\n",
      "tensor([0.0021])\n",
      "tensor([-0.0041])\n",
      "tensor([0.0021])\n",
      "tensor([-0.0040])\n",
      "tensor([0.0020])\n",
      "tensor([-0.0040])\n",
      "tensor([0.0020])\n",
      "tensor([-0.0039])\n",
      "tensor([0.0020])\n",
      "tensor([-0.0038])\n",
      "tensor([0.0019])\n",
      "tensor([-0.0038])\n",
      "tensor([0.0019])\n",
      "tensor([-0.0037])\n",
      "tensor([0.0019])\n",
      "tensor([-0.0037])\n",
      "tensor([0.0019])\n",
      "tensor([-0.0036])\n",
      "tensor([0.0018])\n",
      "tensor([-0.0036])\n",
      "tensor([0.0018])\n",
      "tensor([-0.0035])\n",
      "tensor([0.0018])\n",
      "tensor([-0.0035])\n",
      "tensor([0.0017])\n",
      "tensor([-0.0034])\n",
      "tensor([0.0017])\n",
      "tensor([-0.0034])\n",
      "tensor([0.0017])\n",
      "tensor([-0.0033])\n",
      "tensor([0.0017])\n",
      "tensor([-0.0033])\n",
      "tensor([0.0016])\n",
      "tensor([-0.0032])\n",
      "tensor([0.0016])\n",
      "tensor([-0.0032])\n",
      "tensor([0.0016])\n",
      "tensor([-0.0031])\n",
      "tensor([0.0016])\n",
      "tensor([-0.0031])\n",
      "tensor([0.0015])\n",
      "tensor([-0.0030])\n",
      "tensor([0.0015])\n",
      "tensor([-0.0030])\n",
      "tensor([0.0015])\n",
      "tensor([-0.0029])\n",
      "tensor([0.0015])\n",
      "tensor([-0.0029])\n",
      "tensor([0.0015])\n",
      "tensor([-0.0028])\n",
      "tensor([0.0014])\n",
      "tensor([-0.0028])\n",
      "tensor([0.0014])\n",
      "tensor([-0.0028])\n",
      "tensor([0.0014])\n",
      "tensor([-0.0027])\n",
      "tensor([0.0014])\n",
      "tensor([-0.0027])\n",
      "tensor([0.0013])\n",
      "tensor([-0.0026])\n",
      "tensor([0.0013])\n",
      "tensor([-0.0026])\n",
      "tensor([0.0013])\n",
      "tensor([-0.0026])\n",
      "tensor([0.0013])\n",
      "tensor([-0.0025])\n",
      "tensor([0.0013])\n",
      "tensor([-0.0025])\n",
      "tensor([0.0012])\n",
      "tensor([-0.0024])\n",
      "tensor([0.0012])\n",
      "tensor([-0.0024])\n",
      "tensor([0.0012])\n",
      "tensor([-0.0024])\n",
      "tensor([0.0012])\n",
      "tensor([-0.0023])\n",
      "tensor([0.0012])\n",
      "tensor([-0.0023])\n",
      "tensor([0.0012])\n",
      "tensor([-0.0023])\n",
      "tensor([0.0011])\n",
      "tensor([-0.0022])\n",
      "tensor([0.0011])\n",
      "tensor([-0.0022])\n",
      "tensor([0.0011])\n",
      "tensor([-0.0022])\n",
      "tensor([0.0011])\n",
      "tensor([-0.0021])\n",
      "tensor([0.0011])\n",
      "tensor([-0.0021])\n",
      "tensor([0.0011])\n",
      "tensor([-0.0021])\n",
      "tensor([0.0010])\n",
      "tensor([-0.0020])\n",
      "tensor([0.0010])\n",
      "tensor([-0.0020])\n",
      "tensor([0.0010])\n",
      "tensor([-0.0020])\n",
      "tensor([0.0010])\n",
      "tensor([-0.0019])\n",
      "tensor([0.0010])\n",
      "tensor([-0.0019])\n",
      "tensor([0.0010])\n",
      "tensor([-0.0019])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0019])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0018])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0018])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0018])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0017])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0017])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0017])\n",
      "tensor([0.0009])\n",
      "tensor([-0.0017])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0016])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0016])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0016])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0016])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0015])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0015])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0015])\n",
      "tensor([0.0008])\n",
      "tensor([-0.0015])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0015])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0014])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0014])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0014])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0014])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0013])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0013])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0013])\n",
      "tensor([0.0007])\n",
      "tensor([-0.0013])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0013])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0012])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0012])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0012])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0012])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0012])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0012])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0011])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0011])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0011])\n",
      "tensor([0.0006])\n",
      "tensor([-0.0011])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0011])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0011])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0010])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0010])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0010])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0010])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0010])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0010])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0005])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0009])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0008])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0004])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0007])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0006])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0003])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0005])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0004])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0002])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0003])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([0.0001])\n",
      "tensor([-0.0002])\n",
      "tensor([9.9594e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([9.7900e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([9.6527e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([9.5085e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([9.3636e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([9.2130e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([9.0872e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.9555e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.8250e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.6918e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.5517e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.4095e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.2712e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.1570e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([8.0486e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([7.9145e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([7.8051e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([7.6955e-05])\n",
      "tensor([-0.0002])\n",
      "tensor([7.5681e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([7.4613e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([7.3319e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([7.2196e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([7.1032e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.9903e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.8819e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.7823e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.6842e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.5867e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.4933e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.4093e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.3060e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.2128e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.1231e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([6.0351e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.9320e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.8340e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.7453e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.6731e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.5748e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.4913e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.4107e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.3402e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.2584e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.1776e-05])\n",
      "tensor([-0.0001])\n",
      "tensor([5.1087e-05])\n",
      "tensor([-9.9639e-05])\n",
      "tensor([5.0304e-05])\n",
      "tensor([-9.8126e-05])\n",
      "tensor([4.9497e-05])\n",
      "tensor([-9.6684e-05])\n",
      "tensor([4.8653e-05])\n",
      "tensor([-9.5258e-05])\n",
      "tensor([4.7852e-05])\n",
      "tensor([-9.3863e-05])\n",
      "tensor([4.7217e-05])\n",
      "tensor([-9.2393e-05])\n",
      "tensor([4.6496e-05])\n",
      "tensor([-9.0986e-05])\n",
      "tensor([4.5731e-05])\n",
      "tensor([-8.9653e-05])\n",
      "tensor([4.5108e-05])\n",
      "tensor([-8.8273e-05])\n",
      "tensor([4.4387e-05])\n",
      "tensor([-8.6958e-05])\n",
      "tensor([4.3814e-05])\n",
      "tensor([-8.5594e-05])\n",
      "tensor([4.3082e-05])\n",
      "tensor([-8.4333e-05])\n",
      "tensor([4.2502e-05])\n",
      "tensor([-8.3017e-05])\n",
      "tensor([4.1800e-05])\n",
      "tensor([-8.1784e-05])\n",
      "tensor([4.1258e-05])\n",
      "tensor([-8.0489e-05])\n",
      "tensor([4.0558e-05])\n",
      "tensor([-7.9301e-05])\n",
      "tensor([3.9983e-05])\n",
      "tensor([-7.8079e-05])\n",
      "tensor([3.9203e-05])\n",
      "tensor([-7.6996e-05])\n",
      "tensor([3.8651e-05])\n",
      "tensor([-7.5801e-05])\n",
      "tensor([3.8224e-05])\n",
      "tensor([-7.4577e-05])\n",
      "tensor([3.7707e-05])\n",
      "tensor([-7.3411e-05])\n",
      "tensor([3.7103e-05])\n",
      "tensor([-7.2301e-05])\n",
      "tensor([3.6565e-05])\n",
      "tensor([-7.1188e-05])\n",
      "tensor([3.5924e-05])\n",
      "tensor([-7.0153e-05])\n",
      "tensor([3.5397e-05])\n",
      "tensor([-6.9097e-05])\n",
      "tensor([3.4793e-05])\n",
      "tensor([-6.8070e-05])\n",
      "tensor([3.4280e-05])\n",
      "tensor([-6.7045e-05])\n",
      "tensor([3.3689e-05])\n",
      "tensor([-6.6071e-05])\n",
      "tensor([3.3202e-05])\n",
      "tensor([-6.5076e-05])\n",
      "tensor([3.2708e-05])\n",
      "tensor([-6.4070e-05])\n",
      "tensor([3.2351e-05])\n",
      "tensor([-6.3039e-05])\n",
      "tensor([3.1894e-05])\n",
      "tensor([-6.2071e-05])\n",
      "tensor([3.1338e-05])\n",
      "tensor([-6.1164e-05])\n",
      "tensor([3.0884e-05])\n",
      "tensor([-6.0245e-05])\n",
      "tensor([3.0420e-05])\n",
      "tensor([-5.9323e-05])\n",
      "tensor([2.9829e-05])\n",
      "tensor([-5.8492e-05])\n",
      "tensor([2.9391e-05])\n",
      "tensor([-5.7597e-05])\n",
      "tensor([2.8837e-05])\n",
      "tensor([-5.6787e-05])\n",
      "tensor([2.8545e-05])\n",
      "tensor([-5.5844e-05])\n",
      "tensor([2.8068e-05])\n",
      "tensor([-5.5038e-05])\n",
      "tensor([2.7576e-05])\n",
      "tensor([-5.4227e-05])\n",
      "tensor([2.7157e-05])\n",
      "tensor([-5.3417e-05])\n",
      "tensor([2.6736e-05])\n",
      "tensor([-5.2609e-05])\n",
      "tensor([2.6422e-05])\n",
      "tensor([-5.1769e-05])\n",
      "tensor([2.6061e-05])\n",
      "tensor([-5.0980e-05])\n",
      "tensor([2.5658e-05])\n",
      "tensor([-5.0211e-05])\n",
      "tensor([2.5140e-05])\n",
      "tensor([-4.9515e-05])\n",
      "tensor([2.4863e-05])\n",
      "tensor([-4.8710e-05])\n",
      "tensor([2.4470e-05])\n",
      "tensor([-4.7986e-05])\n",
      "tensor([2.3977e-05])\n",
      "tensor([-4.7319e-05])\n",
      "tensor([2.3683e-05])\n",
      "tensor([-4.6571e-05])\n",
      "tensor([2.3326e-05])\n",
      "tensor([-4.5855e-05])\n",
      "tensor([2.2854e-05])\n",
      "tensor([-4.5239e-05])\n",
      "tensor([2.2599e-05])\n",
      "tensor([-4.4503e-05])\n",
      "tensor([2.2242e-05])\n",
      "tensor([-4.3847e-05])\n",
      "tensor([2.1889e-05])\n",
      "tensor([-4.3183e-05])\n",
      "tensor([2.1613e-05])\n",
      "tensor([-4.2511e-05])\n",
      "tensor([2.1397e-05])\n",
      "tensor([-4.1805e-05])\n",
      "tensor([2.1038e-05])\n",
      "tensor([-4.1201e-05])\n",
      "tensor([2.0693e-05])\n",
      "tensor([-4.0582e-05])\n",
      "tensor([2.0460e-05])\n",
      "tensor([-3.9930e-05])\n",
      "tensor([2.0135e-05])\n",
      "tensor([-3.9340e-05])\n",
      "tensor([1.9830e-05])\n",
      "tensor([-3.8748e-05])\n",
      "tensor([1.9469e-05])\n",
      "tensor([-3.8179e-05])\n",
      "tensor([1.9224e-05])\n",
      "tensor([-3.7587e-05])\n",
      "tensor([1.9010e-05])\n",
      "tensor([-3.6972e-05])\n",
      "tensor([1.8727e-05])\n",
      "tensor([-3.6408e-05])\n",
      "tensor([1.8393e-05])\n",
      "tensor([-3.5875e-05])\n",
      "tensor([1.8167e-05])\n",
      "tensor([-3.5313e-05])\n",
      "tensor([1.7991e-05])\n",
      "tensor([-3.4717e-05])\n",
      "tensor([1.7695e-05])\n",
      "tensor([-3.4217e-05])\n",
      "tensor([1.7414e-05])\n",
      "tensor([-3.3695e-05])\n",
      "tensor([1.6962e-05])\n",
      "tensor([-3.3284e-05])\n",
      "tensor([1.6761e-05])\n",
      "tensor([-3.2756e-05])\n",
      "tensor([1.6481e-05])\n",
      "tensor([-3.2283e-05])\n",
      "tensor([1.6171e-05])\n",
      "tensor([-3.1830e-05])\n",
      "tensor([1.5909e-05])\n",
      "tensor([-3.1345e-05])\n",
      "tensor([1.5728e-05])\n",
      "tensor([-3.0850e-05])\n",
      "tensor([1.5565e-05])\n",
      "tensor([-3.0346e-05])\n",
      "tensor([1.5289e-05])\n",
      "tensor([-2.9915e-05])\n",
      "tensor([1.4985e-05])\n",
      "tensor([-2.9496e-05])\n",
      "tensor([1.4713e-05])\n",
      "tensor([-2.9066e-05])\n",
      "tensor([1.4536e-05])\n",
      "tensor([-2.8615e-05])\n",
      "tensor([1.4409e-05])\n",
      "tensor([-2.8138e-05])\n",
      "tensor([1.4251e-05])\n",
      "tensor([-2.7683e-05])\n",
      "tensor([1.3953e-05])\n",
      "tensor([-2.7311e-05])\n",
      "tensor([1.3675e-05])\n",
      "tensor([-2.6926e-05])\n",
      "tensor([1.3649e-05])\n",
      "tensor([-2.6419e-05])\n",
      "tensor([1.3523e-05])\n",
      "tensor([-2.5994e-05])\n",
      "tensor([1.3353e-05])\n",
      "tensor([-2.5577e-05])\n",
      "tensor([1.3141e-05])\n",
      "tensor([-2.5200e-05])\n",
      "tensor([1.2852e-05])\n",
      "tensor([-2.4875e-05])\n",
      "tensor([1.2607e-05])\n",
      "tensor([-2.4519e-05])\n",
      "tensor([1.2361e-05])\n",
      "tensor([-2.4166e-05])\n",
      "tensor([1.2211e-05])\n",
      "tensor([-2.3796e-05])\n",
      "tensor([1.2074e-05])\n",
      "tensor([-2.3412e-05])\n",
      "tensor([1.1940e-05])\n",
      "tensor([-2.3025e-05])\n",
      "tensor([1.1698e-05])\n",
      "tensor([-2.2717e-05])\n",
      "tensor([1.1454e-05])\n",
      "tensor([-2.2411e-05])\n",
      "tensor([1.1186e-05])\n",
      "tensor([-2.2111e-05])\n",
      "tensor([1.1191e-05])\n",
      "tensor([-2.1685e-05])\n",
      "tensor([1.1090e-05])\n",
      "tensor([-2.1328e-05])\n",
      "tensor([1.0991e-05])\n",
      "tensor([-2.0966e-05])\n",
      "tensor([1.0841e-05])\n",
      "tensor([-2.0638e-05])\n",
      "tensor([1.0597e-05])\n",
      "tensor([-2.0378e-05])\n",
      "tensor([1.0381e-05])\n",
      "tensor([-2.0099e-05])\n",
      "tensor([1.0172e-05])\n",
      "tensor([-1.9811e-05])\n",
      "tensor([9.9134e-06])\n",
      "tensor([-1.9561e-05])\n",
      "tensor([9.8250e-06])\n",
      "tensor([-1.9239e-05])\n",
      "tensor([9.7390e-06])\n",
      "tensor([-1.8921e-05])\n",
      "tensor([9.5970e-06])\n",
      "tensor([-1.8628e-05])\n",
      "tensor([9.5194e-06])\n",
      "tensor([-1.8312e-05])\n",
      "tensor([9.2852e-06])\n",
      "tensor([-1.8094e-05])\n",
      "tensor([9.0717e-06])\n",
      "tensor([-1.7856e-05])\n",
      "tensor([8.8692e-06])\n",
      "tensor([-1.7605e-05])\n",
      "tensor([8.8958e-06])\n",
      "tensor([-1.7261e-05])\n",
      "tensor([8.7665e-06])\n",
      "tensor([-1.7017e-05])\n",
      "tensor([8.7046e-06])\n",
      "tensor([-1.6718e-05])\n",
      "tensor([8.5874e-06])\n",
      "tensor([-1.6478e-05])\n",
      "tensor([8.4943e-06])\n",
      "tensor([-1.6209e-05])\n",
      "tensor([8.4157e-06])\n",
      "tensor([-1.5924e-05])\n",
      "tensor([8.2179e-06])\n",
      "tensor([-1.5733e-05])\n",
      "tensor([8.0168e-06])\n",
      "tensor([-1.5523e-05])\n",
      "tensor([7.8037e-06])\n",
      "tensor([-1.5343e-05])\n",
      "tensor([7.5677e-06])\n",
      "tensor([-1.5163e-05])\n",
      "tensor([7.6334e-06])\n",
      "tensor([-1.4845e-05])\n",
      "tensor([7.5613e-06])\n",
      "tensor([-1.4610e-05])\n",
      "tensor([7.4733e-06])\n",
      "tensor([-1.4378e-05])\n",
      "tensor([7.3778e-06])\n",
      "tensor([-1.4175e-05])\n",
      "tensor([7.3089e-06])\n",
      "tensor([-1.3923e-05])\n",
      "tensor([7.2326e-06])\n",
      "tensor([-1.3692e-05])\n",
      "tensor([7.0194e-06])\n",
      "tensor([-1.3552e-05])\n",
      "tensor([6.8325e-06])\n",
      "tensor([-1.3394e-05])\n",
      "tensor([6.6640e-06])\n",
      "tensor([-1.3220e-05])\n",
      "tensor([6.4909e-06])\n",
      "tensor([-1.3052e-05])\n",
      "tensor([6.4964e-06])\n",
      "tensor([-1.2812e-05])\n",
      "tensor([6.5639e-06])\n",
      "tensor([-1.2530e-05])\n",
      "tensor([6.3789e-06])\n",
      "tensor([-1.2365e-05])\n",
      "tensor([6.2872e-06])\n",
      "tensor([-1.2195e-05])\n",
      "tensor([6.2519e-06])\n",
      "tensor([-1.1981e-05])\n",
      "tensor([6.1895e-06])\n",
      "tensor([-1.1794e-05])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([6.1168e-06])\n",
      "tensor([-1.1602e-05])\n",
      "tensor([6.0374e-06])\n",
      "tensor([-1.1412e-05])\n",
      "tensor([5.9709e-06])\n",
      "tensor([-1.1222e-05])\n",
      "tensor([5.8024e-06])\n",
      "tensor([-1.1109e-05])\n",
      "tensor([5.6330e-06])\n",
      "tensor([-1.0981e-05])\n",
      "tensor([5.4591e-06])\n",
      "tensor([-1.0858e-05])\n",
      "tensor([5.2679e-06])\n",
      "tensor([-1.0748e-05])\n",
      "tensor([5.3230e-06])\n",
      "tensor([-1.0534e-05])\n",
      "tensor([5.3825e-06])\n",
      "tensor([-1.0292e-05])\n",
      "tensor([5.1946e-06])\n",
      "tensor([-1.0184e-05])\n",
      "tensor([5.2743e-06])\n",
      "tensor([-9.9508e-06])\n",
      "tensor([5.2585e-06])\n",
      "tensor([-9.7615e-06])\n",
      "tensor([5.1573e-06])\n",
      "tensor([-9.6367e-06])\n",
      "tensor([5.1380e-06])\n",
      "tensor([-9.4601e-06])\n",
      "tensor([5.0936e-06])\n",
      "tensor([-9.3075e-06])\n",
      "tensor([4.9816e-06])\n",
      "tensor([-9.1954e-06])\n",
      "tensor([4.9298e-06])\n",
      "tensor([-9.0375e-06])\n",
      "tensor([4.8937e-06])\n",
      "tensor([-8.8774e-06])\n",
      "tensor([4.7355e-06])\n",
      "tensor([-8.7903e-06])\n",
      "tensor([4.5751e-06])\n",
      "tensor([-8.7056e-06])\n",
      "tensor([4.4171e-06])\n",
      "tensor([-8.6267e-06])\n",
      "tensor([4.2355e-06])\n",
      "tensor([-8.5554e-06])\n",
      "tensor([4.0668e-06])\n",
      "tensor([-8.4818e-06])\n",
      "tensor([4.1467e-06])\n",
      "tensor([-8.2836e-06])\n",
      "tensor([4.2254e-06])\n",
      "tensor([-8.0930e-06])\n",
      "tensor([4.0760e-06])\n",
      "tensor([-8.0023e-06])\n",
      "tensor([4.1328e-06])\n",
      "tensor([-7.8172e-06])\n",
      "tensor([4.2046e-06])\n",
      "tensor([-7.6315e-06])\n",
      "tensor([3.9215e-06])\n",
      "tensor([-7.6343e-06])\n",
      "tensor([3.8948e-06])\n",
      "tensor([-7.5120e-06])\n",
      "tensor([3.8606e-06])\n",
      "tensor([-7.3902e-06])\n",
      "tensor([3.7850e-06])\n",
      "tensor([-7.3009e-06])\n",
      "tensor([3.7834e-06])\n",
      "tensor([-7.1593e-06])\n",
      "tensor([3.7203e-06])\n",
      "tensor([-7.0526e-06])\n",
      "tensor([3.6638e-06])\n",
      "tensor([-6.9528e-06])\n",
      "tensor([3.6689e-06])\n",
      "tensor([-6.8133e-06])\n",
      "tensor([3.6275e-06])\n",
      "tensor([-6.6917e-06])\n",
      "tensor([3.5784e-06])\n",
      "tensor([-6.5880e-06])\n",
      "tensor([3.5308e-06])\n",
      "tensor([-6.4719e-06])\n",
      "tensor([3.3776e-06])\n",
      "tensor([-6.4364e-06])\n",
      "tensor([3.2437e-06])\n",
      "tensor([-6.3800e-06])\n",
      "tensor([3.0675e-06])\n",
      "tensor([-6.3659e-06])\n",
      "tensor([2.9166e-06])\n",
      "tensor([-6.3259e-06])\n",
      "tensor([3.0048e-06])\n",
      "tensor([-6.1637e-06])\n",
      "tensor([2.8514e-06])\n",
      "tensor([-6.1282e-06])\n",
      "tensor([2.9250e-06])\n",
      "tensor([-5.9746e-06])\n",
      "tensor([3.0333e-06])\n",
      "tensor([-5.8159e-06])\n",
      "tensor([2.9064e-06])\n",
      "tensor([-5.7573e-06])\n",
      "tensor([2.9392e-06])\n",
      "tensor([-5.6362e-06])\n",
      "tensor([3.0205e-06])\n",
      "tensor([-5.4921e-06])\n",
      "tensor([2.9093e-06])\n",
      "tensor([-5.4233e-06])\n",
      "tensor([2.9686e-06])\n",
      "tensor([-5.2866e-06])\n",
      "tensor([2.9444e-06])\n",
      "tensor([-5.2088e-06])\n",
      "tensor([2.9089e-06])\n",
      "tensor([-5.1401e-06])\n",
      "tensor([2.9280e-06])\n",
      "tensor([-5.0343e-06])\n",
      "tensor([2.8831e-06])\n",
      "tensor([-4.9754e-06])\n",
      "tensor([2.8463e-06])\n",
      "tensor([-4.8953e-06])\n",
      "tensor([2.8370e-06])\n",
      "tensor([-4.8150e-06])\n",
      "tensor([2.7919e-06])\n",
      "tensor([-4.7446e-06])\n",
      "tensor([2.7611e-06])\n",
      "tensor([-4.6727e-06])\n",
      "tensor([2.7717e-06])\n",
      "tensor([-4.5722e-06])\n",
      "tensor([2.7131e-06])\n",
      "tensor([-4.5154e-06])\n",
      "tensor([2.6944e-06])\n",
      "tensor([-4.4465e-06])\n",
      "tensor([2.6653e-06])\n",
      "tensor([-4.3539e-06])\n",
      "tensor([2.6430e-06])\n",
      "tensor([-4.2807e-06])\n",
      "tensor([2.6186e-06])\n",
      "tensor([-4.1943e-06])\n",
      "tensor([2.5853e-06])\n",
      "tensor([-4.1222e-06])\n",
      "tensor([2.4510e-06])\n",
      "tensor([-4.1212e-06])\n",
      "tensor([2.3207e-06])\n",
      "tensor([-4.1233e-06])\n",
      "tensor([2.1803e-06])\n",
      "tensor([-4.1167e-06])\n",
      "tensor([2.0407e-06])\n",
      "tensor([-4.1139e-06])\n",
      "tensor([1.9112e-06])\n",
      "tensor([-4.1103e-06])\n",
      "tensor([1.7481e-06])\n",
      "tensor([-4.1249e-06])\n",
      "tensor([1.8650e-06])\n",
      "tensor([-3.9954e-06])\n",
      "tensor([1.7448e-06])\n",
      "tensor([-3.9776e-06])\n",
      "tensor([1.7895e-06])\n",
      "tensor([-3.9126e-06])\n",
      "tensor([1.6539e-06])\n",
      "tensor([-3.9030e-06])\n",
      "tensor([1.7954e-06])\n",
      "tensor([-3.7527e-06])\n",
      "tensor([1.6313e-06])\n",
      "tensor([-3.7805e-06])\n",
      "tensor([1.7231e-06])\n",
      "tensor([-3.6585e-06])\n",
      "tensor([1.8488e-06])\n",
      "tensor([-3.5299e-06])\n",
      "tensor([1.7000e-06])\n",
      "tensor([-3.5376e-06])\n",
      "tensor([1.8062e-06])\n",
      "tensor([-3.4200e-06])\n",
      "tensor([1.6400e-06])\n",
      "tensor([-3.4362e-06])\n",
      "tensor([1.7443e-06])\n",
      "tensor([-3.3185e-06])\n",
      "tensor([1.8582e-06])\n",
      "tensor([-3.1966e-06])\n",
      "tensor([1.7043e-06])\n",
      "tensor([-3.2061e-06])\n",
      "tensor([1.8179e-06])\n",
      "tensor([-3.0808e-06])\n",
      "tensor([1.6859e-06])\n",
      "tensor([-3.0717e-06])\n",
      "tensor([1.7888e-06])\n",
      "tensor([-2.9605e-06])\n",
      "tensor([1.5330e-06])\n",
      "tensor([-3.0368e-06])\n",
      "tensor([1.6415e-06])\n",
      "tensor([-2.9110e-06])\n",
      "tensor([1.6052e-06])\n",
      "tensor([-2.8973e-06])\n",
      "tensor([1.5740e-06])\n",
      "tensor([-2.8721e-06])\n",
      "tensor([1.6280e-06])\n",
      "tensor([-2.7748e-06])\n",
      "tensor([1.6018e-06])\n",
      "tensor([-2.7442e-06])\n",
      "tensor([1.5683e-06])\n",
      "tensor([-2.7200e-06])\n",
      "tensor([1.5324e-06])\n",
      "tensor([-2.7094e-06])\n",
      "tensor([1.5143e-06])\n",
      "tensor([-2.6750e-06])\n",
      "tensor([1.5338e-06])\n",
      "tensor([-2.6159e-06])\n",
      "tensor([1.5271e-06])\n",
      "tensor([-2.5660e-06])\n",
      "tensor([1.5131e-06])\n",
      "tensor([-2.5269e-06])\n",
      "tensor([1.4782e-06])\n",
      "tensor([-2.5077e-06])\n",
      "tensor([1.5043e-06])\n",
      "tensor([-2.4402e-06])\n",
      "tensor([1.4775e-06])\n",
      "tensor([-2.4112e-06])\n",
      "tensor([1.4729e-06])\n",
      "tensor([-2.3637e-06])\n",
      "tensor([1.4377e-06])\n",
      "tensor([-2.3390e-06])\n",
      "tensor([1.4158e-06])\n",
      "tensor([-2.3076e-06])\n",
      "tensor([1.4075e-06])\n",
      "tensor([-2.2663e-06])\n",
      "tensor([1.3832e-06])\n",
      "tensor([-2.2354e-06])\n",
      "tensor([1.3728e-06])\n",
      "tensor([-2.1949e-06])\n",
      "tensor([1.3587e-06])\n",
      "tensor([-2.1585e-06])\n",
      "tensor([1.3668e-06])\n",
      "tensor([-2.1037e-06])\n",
      "tensor([1.3468e-06])\n",
      "tensor([-2.0684e-06])\n",
      "tensor([1.3252e-06])\n",
      "tensor([-2.0346e-06])\n",
      "tensor([1.2870e-06])\n",
      "tensor([-2.0225e-06])\n",
      "tensor([1.3121e-06])\n",
      "tensor([-1.9546e-06])\n",
      "tensor([1.2775e-06])\n",
      "tensor([-1.9293e-06])\n",
      "tensor([1.2558e-06])\n",
      "tensor([-1.8950e-06])\n",
      "tensor([1.2472e-06])\n",
      "tensor([-1.8566e-06])\n",
      "tensor([1.2209e-06])\n",
      "tensor([-1.8266e-06])\n",
      "tensor([1.2135e-06])\n",
      "tensor([-1.7829e-06])\n",
      "tensor([1.1082e-06])\n",
      "tensor([-1.8117e-06])\n",
      "tensor([1.1006e-06])\n",
      "tensor([-1.7641e-06])\n",
      "tensor([9.7632e-07])\n",
      "tensor([-1.7986e-06])\n",
      "tensor([9.7539e-07])\n",
      "tensor([-1.7582e-06])\n",
      "tensor([8.0356e-07])\n",
      "tensor([-1.8217e-06])\n",
      "tensor([8.1823e-07])\n",
      "tensor([-1.7640e-06])\n",
      "tensor([6.8819e-07])\n",
      "tensor([-1.8052e-06])\n",
      "tensor([6.7143e-07])\n",
      "tensor([-1.7707e-06])\n",
      "tensor([5.7038e-07])\n",
      "tensor([-1.7968e-06])\n",
      "tensor([7.9436e-07])\n",
      "tensor([-1.6379e-06])\n",
      "tensor([6.8540e-07])\n",
      "tensor([-1.6615e-06])\n",
      "tensor([5.5338e-07])\n",
      "tensor([-1.6991e-06])\n",
      "tensor([6.2975e-07])\n",
      "tensor([-1.6509e-06])\n",
      "tensor([5.2428e-07])\n",
      "tensor([-1.6757e-06])\n",
      "tensor([6.0367e-07])\n",
      "tensor([-1.6287e-06])\n",
      "tensor([5.0268e-07])\n",
      "tensor([-1.6558e-06])\n",
      "tensor([5.9878e-07])\n",
      "tensor([-1.5878e-06])\n",
      "tensor([4.6077e-07])\n",
      "tensor([-1.6291e-06])\n",
      "tensor([6.1322e-07])\n",
      "tensor([-1.5286e-06])\n",
      "tensor([4.5472e-07])\n",
      "tensor([-1.5798e-06])\n",
      "tensor([5.9442e-07])\n",
      "tensor([-1.4876e-06])\n",
      "tensor([7.1566e-07])\n",
      "tensor([-1.4041e-06])\n",
      "tensor([5.7905e-07])\n",
      "tensor([-1.4518e-06])\n",
      "tensor([7.1316e-07])\n",
      "tensor([-1.3691e-06])\n",
      "tensor([5.7695e-07])\n",
      "tensor([-1.4135e-06])\n",
      "tensor([6.8895e-07])\n",
      "tensor([-1.3313e-06])\n",
      "tensor([5.7020e-07])\n",
      "tensor([-1.3629e-06])\n",
      "tensor([6.6496e-07])\n",
      "tensor([-1.3030e-06])\n",
      "tensor([5.4319e-07])\n",
      "tensor([-1.3332e-06])\n",
      "tensor([6.3772e-07])\n",
      "tensor([-1.2715e-06])\n",
      "tensor([5.2160e-07])\n",
      "tensor([-1.2996e-06])\n",
      "tensor([6.2003e-07])\n",
      "tensor([-1.2364e-06])\n",
      "tensor([5.0204e-07])\n",
      "tensor([-1.2649e-06])\n",
      "tensor([6.1165e-07])\n",
      "tensor([-1.1917e-06])\n",
      "tensor([4.8836e-07])\n",
      "tensor([-1.2334e-06])\n",
      "tensor([6.0169e-07])\n",
      "tensor([-1.1534e-06])\n",
      "tensor([4.7975e-07])\n",
      "tensor([-1.1903e-06])\n",
      "tensor([6.0309e-07])\n",
      "tensor([-1.1031e-06])\n",
      "tensor([4.7206e-07])\n",
      "tensor([-1.1449e-06])\n",
      "tensor([5.8918e-07])\n",
      "tensor([-1.0631e-06])\n",
      "tensor([6.9622e-07])\n",
      "tensor([-1.0042e-06])\n",
      "tensor([5.5681e-07])\n",
      "tensor([-1.0437e-06])\n",
      "tensor([6.6601e-07])\n",
      "tensor([-9.7105e-07])\n",
      "tensor([5.3446e-07])\n",
      "tensor([-1.0067e-06])\n",
      "tensor([6.4389e-07])\n",
      "tensor([-9.3217e-07])\n",
      "tensor([5.2887e-07])\n",
      "tensor([-9.6860e-07])\n",
      "tensor([6.5553e-07])\n",
      "tensor([-8.7361e-07])\n",
      "tensor([5.4983e-07])\n",
      "tensor([-9.0632e-07])\n",
      "tensor([6.3831e-07])\n",
      "tensor([-8.4148e-07])\n",
      "tensor([5.1101e-07])\n",
      "tensor([-8.8318e-07])\n",
      "tensor([6.2899e-07])\n",
      "tensor([-8.0283e-07])\n",
      "tensor([5.2544e-07])\n",
      "tensor([-8.2533e-07])\n",
      "tensor([6.2108e-07])\n",
      "tensor([-7.6744e-07])\n",
      "tensor([4.7981e-07])\n",
      "tensor([-8.0618e-07])\n",
      "tensor([5.8505e-07])\n",
      "tensor([-7.2928e-07])\n",
      "tensor([7.1654e-07])\n",
      "tensor([-6.4241e-07])\n",
      "tensor([5.8761e-07])\n",
      "tensor([-6.8365e-07])\n",
      "tensor([6.7259e-07])\n",
      "tensor([-6.2917e-07])\n",
      "tensor([5.8272e-07])\n",
      "tensor([-6.5076e-07])\n",
      "tensor([6.8586e-07])\n",
      "tensor([-5.7608e-07])\n",
      "tensor([4.5373e-07])\n",
      "tensor([-6.8860e-07])\n",
      "tensor([5.7620e-07])\n",
      "tensor([-6.0361e-07])\n",
      "tensor([6.7911e-07])\n",
      "tensor([-5.3330e-07])\n",
      "tensor([4.4022e-07])\n",
      "tensor([-6.4343e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([5.4640e-07])\n",
      "tensor([-5.7230e-07])\n",
      "tensor([1.0235], requires_grad=True) tensor([1.9690], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "lr = 1e-1\n",
    "n_epochs = 1000\n",
    "\n",
    "torch.manual_seed(42)\n",
    "a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    yhat = a + b * x_train_tensor\n",
    "    error = y_train_tensor - yhat\n",
    "    loss = (error ** 2).mean()\n",
    "\n",
    "    # No more manual computation of gradients! \n",
    "    # a_grad = -2 * error.mean()\n",
    "    # b_grad = -2 * (x_tensor * error).mean()\n",
    "    \n",
    "    # We just tell PyTorch to work its way BACKWARDS from the specified loss!\n",
    "    loss.backward()\n",
    "    # Let's check the computed gradients...\n",
    "    print(a.grad)\n",
    "    print(b.grad)\n",
    "    \n",
    "    # What about UPDATING the parameters? Not so fast...\n",
    "    \n",
    "    # FIRST ATTEMPT\n",
    "    # AttributeError: 'NoneType' object has no attribute 'zero_'\n",
    "    # a = a - lr * a.grad\n",
    "    # b = b - lr * b.grad\n",
    "    # print(a)\n",
    "\n",
    "    # SECOND ATTEMPT\n",
    "    # RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.\n",
    "    # a -= lr * a.grad\n",
    "    # b -= lr * b.grad        \n",
    "    \n",
    "    # THIRD ATTEMPT\n",
    "    # We need to use NO_GRAD to keep the update out of the gradient computation\n",
    "    # Why is that? It boils down to the DYNAMIC GRAPH that PyTorch uses...\n",
    "    with torch.no_grad():\n",
    "        a -= lr * a.grad\n",
    "        b -= lr * b.grad\n",
    "    \n",
    "    # PyTorch is \"clingy\" to its computed gradients, we need to tell it to let it go...\n",
    "    a.grad.zero_()\n",
    "    b.grad.zero_()\n",
    "    \n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dynamic Computation Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:00:00.918574Z",
     "start_time": "2019-06-19T11:00:00.908904Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(42)\n",
    "a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "\n",
    "yhat = a + b * x_train_tensor\n",
    "error = y_train_tensor - yhat\n",
    "loss = (error ** 2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:03:50.804137Z",
     "start_time": "2019-06-19T11:03:50.684759Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](./img/neuralnetwork/comgraph.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:00:01.599960Z",
     "start_time": "2019-06-19T11:00:01.564001Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"170pt\" height=\"168pt\"\n",
       " viewBox=\"0.00 0.00 170.32 168.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 164)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-164 166.3242,-164 166.3242,4 -4,4\"/>\n",
       "<!-- 112483094088 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>112483094088</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"#000000\" points=\"117.8111,-20 26.1889,-20 26.1889,0 117.8111,0 117.8111,-20\"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-6.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 112483093696 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>112483093696</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"54,-90 0,-90 0,-56 54,-56 54,-90\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-62.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 112483093696&#45;&gt;112483094088 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>112483093696&#45;&gt;112483094088</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M39.2964,-55.785C45.3224,-47.3487 52.587,-37.1781 58.7374,-28.5676\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"61.6804,-30.4691 64.6447,-20.2974 55.9842,-26.4004 61.6804,-30.4691\"/>\n",
       "</g>\n",
       "<!-- 112483094312 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>112483094312</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"162.1491,-83 71.8509,-83 71.8509,-63 162.1491,-63 162.1491,-83\"/>\n",
       "<text text-anchor=\"middle\" x=\"117\" y=\"-69.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">MulBackward0</text>\n",
       "</g>\n",
       "<!-- 112483094312&#45;&gt;112483094088 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>112483094312&#45;&gt;112483094088</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M109.77,-62.878C103.183,-53.6563 93.2902,-39.8063 85.2814,-28.594\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"88.0221,-26.4092 79.3616,-20.3062 82.3259,-30.4779 88.0221,-26.4092\"/>\n",
       "</g>\n",
       "<!-- 112483092240 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>112483092240</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"144,-160 90,-160 90,-126 144,-126 144,-160\"/>\n",
       "<text text-anchor=\"middle\" x=\"117\" y=\"-132.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 112483092240&#45;&gt;112483094312 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>112483092240&#45;&gt;112483094312</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M117,-125.6966C117,-115.9445 117,-103.7049 117,-93.4768\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"120.5001,-93.3805 117,-83.3805 113.5001,-93.3806 120.5001,-93.3805\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1a3083cba8>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_dot(yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:08:28.517406Z",
     "start_time": "2019-06-19T11:08:28.482259Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"170pt\" height=\"224pt\"\n",
       " viewBox=\"0.00 0.00 170.32 224.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 220)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-220 166.3242,-220 166.3242,4 -4,4\"/>\n",
       "<!-- 112421801376 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>112421801376</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"#000000\" points=\"116.8189,-20 27.1811,-20 27.1811,0 116.8189,0 116.8189,-20\"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-6.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SubBackward0</text>\n",
       "</g>\n",
       "<!-- 112483094088 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>112483094088</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"117.8111,-76 26.1889,-76 26.1889,-56 117.8111,-56 117.8111,-76\"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-62.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 112483094088&#45;&gt;112421801376 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>112483094088&#45;&gt;112421801376</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M72,-55.9883C72,-48.9098 72,-39.1714 72,-30.4779\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"75.5001,-30.3038 72,-20.3039 68.5001,-30.3039 75.5001,-30.3038\"/>\n",
       "</g>\n",
       "<!-- 112483093696 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>112483093696</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"54,-146 0,-146 0,-112 54,-112 54,-146\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-118.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 112483093696&#45;&gt;112483094088 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>112483093696&#45;&gt;112483094088</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M39.2964,-111.785C45.3224,-103.3487 52.587,-93.1781 58.7374,-84.5676\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"61.6804,-86.4691 64.6447,-76.2974 55.9842,-82.4004 61.6804,-86.4691\"/>\n",
       "</g>\n",
       "<!-- 112483094312 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>112483094312</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"162.1491,-139 71.8509,-139 71.8509,-119 162.1491,-119 162.1491,-139\"/>\n",
       "<text text-anchor=\"middle\" x=\"117\" y=\"-125.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">MulBackward0</text>\n",
       "</g>\n",
       "<!-- 112483094312&#45;&gt;112483094088 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>112483094312&#45;&gt;112483094088</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M109.77,-118.878C103.183,-109.6563 93.2902,-95.8063 85.2814,-84.594\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"88.0221,-82.4092 79.3616,-76.3062 82.3259,-86.4779 88.0221,-82.4092\"/>\n",
       "</g>\n",
       "<!-- 112483092240 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>112483092240</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"144,-216 90,-216 90,-182 144,-182 144,-216\"/>\n",
       "<text text-anchor=\"middle\" x=\"117\" y=\"-188.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 112483092240&#45;&gt;112483094312 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>112483092240&#45;&gt;112483094312</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M117,-181.6966C117,-171.9445 117,-159.7049 117,-149.4768\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"120.5001,-149.3805 117,-139.3805 113.5001,-149.3806 120.5001,-149.3805\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1a2cdc8fd0>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_dot(error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:00:02.726832Z",
     "start_time": "2019-06-19T11:00:02.691818Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"170pt\" height=\"336pt\"\n",
       " viewBox=\"0.00 0.00 170.32 336.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 332)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-332 166.3242,-332 166.3242,4 -4,4\"/>\n",
       "<!-- 112483094368 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>112483094368</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"#000000\" points=\"120.9668,-20 23.0332,-20 23.0332,0 120.9668,0 120.9668,-20\"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-6.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">MeanBackward1</text>\n",
       "</g>\n",
       "<!-- 4728385608 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>4728385608</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"117.9844,-76 26.0156,-76 26.0156,-56 117.9844,-56 117.9844,-76\"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-62.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">PowBackward0</text>\n",
       "</g>\n",
       "<!-- 4728385608&#45;&gt;112483094368 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>4728385608&#45;&gt;112483094368</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M72,-55.9883C72,-48.9098 72,-39.1714 72,-30.4779\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"75.5001,-30.3038 72,-20.3039 68.5001,-30.3039 75.5001,-30.3038\"/>\n",
       "</g>\n",
       "<!-- 112421801376 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>112421801376</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"116.8189,-132 27.1811,-132 27.1811,-112 116.8189,-112 116.8189,-132\"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-118.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">SubBackward0</text>\n",
       "</g>\n",
       "<!-- 112421801376&#45;&gt;4728385608 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>112421801376&#45;&gt;4728385608</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M72,-111.9883C72,-104.9098 72,-95.1714 72,-86.4779\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"75.5001,-86.3038 72,-76.3039 68.5001,-86.3039 75.5001,-86.3038\"/>\n",
       "</g>\n",
       "<!-- 112483094088 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>112483094088</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"117.8111,-188 26.1889,-188 26.1889,-168 117.8111,-168 117.8111,-188\"/>\n",
       "<text text-anchor=\"middle\" x=\"72\" y=\"-174.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddBackward0</text>\n",
       "</g>\n",
       "<!-- 112483094088&#45;&gt;112421801376 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>112483094088&#45;&gt;112421801376</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M72,-167.9883C72,-160.9098 72,-151.1714 72,-142.4779\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"75.5001,-142.3038 72,-132.3039 68.5001,-142.3039 75.5001,-142.3038\"/>\n",
       "</g>\n",
       "<!-- 112483093696 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>112483093696</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"54,-258 0,-258 0,-224 54,-224 54,-258\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-230.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 112483093696&#45;&gt;112483094088 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>112483093696&#45;&gt;112483094088</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M39.2964,-223.785C45.3224,-215.3487 52.587,-205.1781 58.7374,-196.5676\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"61.6804,-198.4691 64.6447,-188.2974 55.9842,-194.4004 61.6804,-198.4691\"/>\n",
       "</g>\n",
       "<!-- 112483094312 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>112483094312</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"162.1491,-251 71.8509,-251 71.8509,-231 162.1491,-231 162.1491,-251\"/>\n",
       "<text text-anchor=\"middle\" x=\"117\" y=\"-237.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">MulBackward0</text>\n",
       "</g>\n",
       "<!-- 112483094312&#45;&gt;112483094088 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>112483094312&#45;&gt;112483094088</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M109.77,-230.878C103.183,-221.6563 93.2902,-207.8063 85.2814,-196.594\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"88.0221,-194.4092 79.3616,-188.3062 82.3259,-198.4779 88.0221,-194.4092\"/>\n",
       "</g>\n",
       "<!-- 112483092240 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>112483092240</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"144,-328 90,-328 90,-294 144,-294 144,-328\"/>\n",
       "<text text-anchor=\"middle\" x=\"117\" y=\"-300.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 112483092240&#45;&gt;112483094312 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>112483092240&#45;&gt;112483094312</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M117,-293.6966C117,-283.9445 117,-271.7049 117,-261.4768\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"120.5001,-261.3805 117,-251.3805 113.5001,-261.3806 120.5001,-261.3805\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1a3083cc88>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_dot(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Optimizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:23:12.135653Z",
     "start_time": "2019-06-19T08:23:11.941223Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.3367], requires_grad=True) tensor([0.1288], requires_grad=True)\n",
      "tensor([1.0235], requires_grad=True) tensor([1.9690], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "print(a, b)\n",
    "\n",
    "lr = 1e-1\n",
    "n_epochs = 1000\n",
    "\n",
    "# Defines a SGD optimizer to update the parameters\n",
    "optimizer = optim.SGD([a, b], lr=lr)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    yhat = a + b * x_train_tensor\n",
    "    error = y_train_tensor - yhat\n",
    "    loss = (error ** 2).mean()\n",
    "\n",
    "    loss.backward()    \n",
    "    \n",
    "    # No more manual update!\n",
    "    # with torch.no_grad():\n",
    "    #     a -= lr * a.grad\n",
    "    #     b -= lr * b.grad\n",
    "    optimizer.step()\n",
    "    \n",
    "    # No more telling PyTorch to let gradients go!\n",
    "    # a.grad.zero_()\n",
    "    # b.grad.zero_()\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T08:23:13.455326Z",
     "start_time": "2019-06-19T08:23:13.254338Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.3367], requires_grad=True) tensor([0.1288], requires_grad=True)\n",
      "tensor([1.0235], requires_grad=True) tensor([1.9690], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)\n",
    "print(a, b)\n",
    "\n",
    "lr = 1e-1\n",
    "n_epochs = 1000\n",
    "\n",
    "# Defines a MSE loss function\n",
    "loss_fn = nn.MSELoss(reduction='mean')\n",
    "# optimizer in action!\n",
    "optimizer = optim.SGD([a, b], lr=lr)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    yhat = a + b * x_train_tensor\n",
    "    \n",
    "    # No more manual loss!\n",
    "    # error = y_tensor - yhat\n",
    "    # loss = (error ** 2).mean()\n",
    "    loss = loss_fn(y_train_tensor, yhat)\n",
    "\n",
    "    loss.backward()    \n",
    "    optimizer.step()\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "print(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:09.545334Z",
     "start_time": "2019-06-19T11:21:09.535020Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ManualLinearRegression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # To make \"a\" and \"b\" real parameters of the model, we need to wrap them with nn.Parameter\n",
    "        self.a = nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float))\n",
    "        self.b = nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float))\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # Computes the outputs / predictions\n",
    "        return self.a + self.b * x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:10.431066Z",
     "start_time": "2019-06-19T11:21:10.214405Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('a', tensor([0.3367])), ('b', tensor([0.1288]))])\n",
      "OrderedDict([('a', tensor([1.0235])), ('b', tensor([1.9690]))])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(42)\n",
    "\n",
    "# Now we can create a model and send it at once to the device\n",
    "model = ManualLinearRegression().to(device)\n",
    "# We can also inspect its parameters using its state_dict\n",
    "print(model.state_dict())\n",
    "\n",
    "lr = 1e-1\n",
    "n_epochs = 1000\n",
    "\n",
    "loss_fn = nn.MSELoss(reduction='mean')\n",
    "optimizer = optim.SGD(model.parameters(), lr=lr)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    # What is this?!?\n",
    "    model.train()\n",
    "\n",
    "    # No more manual prediction!\n",
    "    # yhat = a + b * x_tensor\n",
    "    yhat = model(x_train_tensor)\n",
    "    \n",
    "    loss = loss_fn(y_train_tensor, yhat)\n",
    "    loss.backward()    \n",
    "    optimizer.step()\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "print(model.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Nested Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:12.144733Z",
     "start_time": "2019-06-19T11:21:12.136576Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class LayerLinearRegression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # Instead of our custom parameters, we use a Linear layer \n",
    "        # with single input and single output\n",
    "        self.linear = nn.Linear(1, 1)\n",
    "                \n",
    "    def forward(self, x):\n",
    "        # Now it only takes a call to the layer to make predictions\n",
    "        return self.linear(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sequential Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:13.329035Z",
     "start_time": "2019-06-19T11:21:13.325189Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Alternatively, you can use a Sequential model\n",
    "model = nn.Sequential(nn.Linear(1, 1)).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:14.674516Z",
     "start_time": "2019-06-19T11:21:14.423316Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('0.weight', tensor([[-0.2191]])), ('0.bias', tensor([0.2018]))])\n"
     ]
    }
   ],
   "source": [
    "def make_train_step(model, loss_fn, optimizer):\n",
    "    # Builds function that performs a step in the train loop\n",
    "    def train_step(x, y):\n",
    "        # Sets model to TRAIN mode\n",
    "        model.train()\n",
    "        # Makes predictions\n",
    "        yhat = model(x)\n",
    "        # Computes loss\n",
    "        loss = loss_fn(y, yhat)\n",
    "        # Computes gradients\n",
    "        loss.backward()\n",
    "        # Updates parameters and zeroes gradients\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "        # Returns the loss\n",
    "        return loss.item()\n",
    "    \n",
    "    # Returns the function that will be called inside the train loop\n",
    "    return train_step\n",
    "\n",
    "# Creates the train_step function for our model, loss function and optimizer\n",
    "train_step = make_train_step(model, loss_fn, optimizer)\n",
    "losses = []\n",
    "\n",
    "# For each epoch...\n",
    "for epoch in range(n_epochs):\n",
    "    # Performs one train step and returns the corresponding loss\n",
    "    loss = train_step(x_train_tensor, y_train_tensor)\n",
    "    losses.append(loss)\n",
    "    \n",
    "# Checks model's parameters\n",
    "print(model.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:16.919397Z",
     "start_time": "2019-06-19T11:21:16.893935Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(tensor([0.7713]), tensor([2.4745]))\n",
      "(tensor([0.7713]), tensor([2.4745]))\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.data import Dataset, TensorDataset\n",
    "\n",
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, x_tensor, y_tensor):\n",
    "        self.x = x_tensor\n",
    "        self.y = y_tensor\n",
    "        \n",
    "    def __getitem__(self, index):\n",
    "        return (self.x[index], self.y[index])\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.x)\n",
    "\n",
    "# Wait, is this a CPU tensor now? Why? Where is .to(device)?\n",
    "x_train_tensor = torch.from_numpy(x_train).float()\n",
    "y_train_tensor = torch.from_numpy(y_train).float()\n",
    "\n",
    "train_data = CustomDataset(x_train_tensor, y_train_tensor)\n",
    "print(train_data[0])\n",
    "\n",
    "train_data = TensorDataset(x_train_tensor, y_train_tensor)\n",
    "print(train_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:18.391907Z",
     "start_time": "2019-06-19T11:21:18.387753Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "train_loader = DataLoader(dataset=train_data, batch_size=16, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:18.916313Z",
     "start_time": "2019-06-19T11:21:18.910055Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([[0.4561],\n",
       "         [0.3745],\n",
       "         [0.1987],\n",
       "         [0.7608],\n",
       "         [0.1196],\n",
       "         [0.1997],\n",
       "         [0.7751],\n",
       "         [0.2713],\n",
       "         [0.6233],\n",
       "         [0.9699],\n",
       "         [0.0452],\n",
       "         [0.0254],\n",
       "         [0.8662],\n",
       "         [0.7081],\n",
       "         [0.8872],\n",
       "         [0.1560]]), tensor([[1.7706],\n",
       "         [1.7578],\n",
       "         [1.2654],\n",
       "         [2.4970],\n",
       "         [1.3214],\n",
       "         [1.3651],\n",
       "         [2.4936],\n",
       "         [1.5105],\n",
       "         [2.2940],\n",
       "         [2.9727],\n",
       "         [0.9985],\n",
       "         [1.0785],\n",
       "         [2.6805],\n",
       "         [2.3660],\n",
       "         [2.8708],\n",
       "         [1.2901]])]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(train_loader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:21:21.413869Z",
     "start_time": "2019-06-19T11:21:19.467745Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('0.weight', tensor([[-0.2191]])), ('0.bias', tensor([0.2018]))])\n"
     ]
    }
   ],
   "source": [
    "losses = []\n",
    "train_step = make_train_step(model, loss_fn, optimizer)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for x_batch, y_batch in train_loader:\n",
    "        # the dataset \"lives\" in the CPU, so do our mini-batches\n",
    "        # therefore, we need to send those mini-batches to the\n",
    "        # device where the model \"lives\"\n",
    "        x_batch = x_batch.to(device)\n",
    "        y_batch = y_batch.to(device)\n",
    "        \n",
    "        loss = train_step(x_batch, y_batch)\n",
    "        losses.append(loss)\n",
    "        \n",
    "print(model.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Random Split\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:23:07.013796Z",
     "start_time": "2019-06-19T11:23:07.004064Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data.dataset import random_split\n",
    "\n",
    "x_tensor = torch.from_numpy(x).float()\n",
    "y_tensor = torch.from_numpy(y).float()\n",
    "\n",
    "dataset = TensorDataset(x_tensor, y_tensor)\n",
    "\n",
    "train_dataset, val_dataset = random_split(dataset, [80, 20])\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, batch_size=16)\n",
    "val_loader = DataLoader(dataset=val_dataset, batch_size=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:23:38.430475Z",
     "start_time": "2019-06-19T11:23:35.756714Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('0.weight', tensor([[-0.2191]])), ('0.bias', tensor([0.2018]))])\n"
     ]
    }
   ],
   "source": [
    "losses = []\n",
    "val_losses = []\n",
    "train_step = make_train_step(model, loss_fn, optimizer)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for x_batch, y_batch in train_loader:\n",
    "        x_batch = x_batch.to(device)\n",
    "        y_batch = y_batch.to(device)\n",
    "\n",
    "        loss = train_step(x_batch, y_batch)\n",
    "        losses.append(loss)\n",
    "        \n",
    "    with torch.no_grad():\n",
    "        for x_val, y_val in val_loader:\n",
    "            x_val = x_val.to(device)\n",
    "            y_val = y_val.to(device)\n",
    "            \n",
    "            model.eval()\n",
    "\n",
    "            yhat = model(x_val)\n",
    "            val_loss = loss_fn(y_val, yhat)\n",
    "            val_losses.append(val_loss.item())\n",
    "\n",
    "print(model.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## All together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:24:20.777508Z",
     "start_time": "2019-06-19T11:24:20.234807Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('0.weight', tensor([[-0.9676]])), ('0.bias', tensor([-0.5727]))])\n",
      "OrderedDict([('0.weight', tensor([[1.9625]])), ('0.bias', tensor([1.0147]))])\n",
      "0.048798722923422855\n",
      "0.020732787313560645\n"
     ]
    }
   ],
   "source": [
    "# https://gist.github.com/dvgodoy/1d818d86a6a0dc6e7c07610835b46fe4\n",
    "torch.manual_seed(42)\n",
    "\n",
    "x_tensor = torch.from_numpy(x).float()\n",
    "y_tensor = torch.from_numpy(y).float()\n",
    "\n",
    "# Builds dataset with ALL data\n",
    "dataset = TensorDataset(x_tensor, y_tensor)\n",
    "# Splits randomly into train and validation datasets\n",
    "train_dataset, val_dataset = random_split(dataset, [80, 20])\n",
    "# Builds a loader for each dataset to perform mini-batch gradient descent\n",
    "train_loader = DataLoader(dataset=train_dataset, batch_size=16)\n",
    "val_loader = DataLoader(dataset=val_dataset, batch_size=20)\n",
    "\n",
    "# Builds a simple sequential model\n",
    "model = nn.Sequential(nn.Linear(1, 1)).to(device)\n",
    "print(model.state_dict())\n",
    "\n",
    "# Sets hyper-parameters\n",
    "lr = 1e-1\n",
    "n_epochs = 150\n",
    "\n",
    "# Defines loss function and optimizer\n",
    "loss_fn = nn.MSELoss(reduction='mean')\n",
    "optimizer = optim.SGD(model.parameters(), lr=lr)\n",
    "\n",
    "losses = []\n",
    "val_losses = []\n",
    "# Creates function to perform train step from model, loss and optimizer\n",
    "train_step = make_train_step(model, loss_fn, optimizer)\n",
    "\n",
    "# Training loop\n",
    "for epoch in range(n_epochs):\n",
    "    # Uses loader to fetch one mini-batch for training\n",
    "    for x_batch, y_batch in train_loader:\n",
    "        # NOW, sends the mini-batch data to the device\n",
    "        # so it matches location of the MODEL\n",
    "        x_batch = x_batch.to(device)\n",
    "        y_batch = y_batch.to(device)\n",
    "        # One stpe of training\n",
    "        loss = train_step(x_batch, y_batch)\n",
    "        losses.append(loss)\n",
    "        \n",
    "    # After finishing training steps for all mini-batches,\n",
    "    # it is time for evaluation!\n",
    "        \n",
    "    # We tell PyTorch to NOT use autograd...\n",
    "    # Do you remember why?\n",
    "    with torch.no_grad():\n",
    "        # Uses loader to fetch one mini-batch for validation\n",
    "        for x_val, y_val in val_loader:\n",
    "            # Again, sends data to same device as model\n",
    "            x_val = x_val.to(device)\n",
    "            y_val = y_val.to(device)\n",
    "            \n",
    "            # What is that?!\n",
    "            model.eval()\n",
    "            # Makes predictions\n",
    "            yhat = model(x_val)\n",
    "            # Computes validation loss\n",
    "            val_loss = loss_fn(y_val, yhat)\n",
    "            val_losses.append(val_loss.item())\n",
    "\n",
    "print(model.state_dict())\n",
    "print(np.mean(losses))\n",
    "print(np.mean(val_losses))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-19T11:29:40.032800Z",
     "start_time": "2019-06-19T11:29:39.845884Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD+CAYAAADF/ZVnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAFfdJREFUeJzt3X+03HV95/Hne+7khvySxOSGYAJFoIABCmh2y1K72qLEUg4e7RZtt9YtrZzd427ZLtrds6iH7fF4uit4rFrbUmrTurS1q4tdqmxBWTnVBSURFlsUjYFAQq6EEMnvkHvve//4zg3DvTNzR7g3M5/J83FOzp35fr+ZeWdy87rvfL6f7+cbmYkkaTDUel2AJGn2GOqSNEAMdUkaIIa6JA0QQ12SBoihLkkDxFCXpAFiqEvSADHUJWmA1I/1G65YsSJPO+20Y/22klS0TZs2PZ2ZIzMdd8xD/bTTTmPjxo3H+m0lqWgRsbWb4xx+kaQBYqhL0gAx1CVpgBjqkjRADHVJGiCGuiQNEENdkgZIMaG+49mDfOTOR9iyc1+vS5GkvlVMqD+15zAfu3szj+3a3+tSJKlvFRPqQ7UAYGzcG2VLUjvFhfpEGuqS1E5xoT42YahLUjvFhfq4oS5JbZUT6mGoS9JMygl1O3VJmpGhLkkDpJhQr0+GurNfJKmtrkI9Iq6KiEcjYnNEXD1l3xsi4luN/Z+KiKE5KdROXZJmNGOoR8QS4CbgtY1fH4qI5vvk/THwS8CZwFnAZXNQ5/OduqEuSW1106mvB+7JzO2ZOQrcDVzatP9w0+P5wFOzWN9RduqSNLNubjx9CtB8w9NtwMlNz38F+AxwEPh0Zm6a+gIRcQ1wDcCpp5764go11CVpRt106sPARNPzCWC86fm7gD8AfhP4hYiYltqZeXNmrsvMdSMjI1N3d1doeEWpJM2km1DfAaxuer4GeAIgItYCr8nMT2Tm/wU+D7x71qukae0XQ12S2uom1O8E1kfEyohYBVzS2AbVePqpEXFSRNSAi4Ddc1HokJ26JM1oxjH1zByNiOuBexubrgMui4gzMvPGiPhvwP1UAX8/8LG5KLRWCyJcpVGSOunmRCmZuQHY0GbfjcCNs1dSe/Va2KlLUgfFXFEK1clSx9Qlqb2iQt1OXZI6KyrUa7VwnrokdVBUqNdr4YlSSeqgqFAfcvhFkjoqLtQ9USpJ7ZUV6mGnLkmdlBXqQ3bqktRJWaFupy5JHZUV6rXwdnaS1EF5oT5uqEtSO4WFes1OXZI6KCzUvfORJHVSWKjXDHVJ6qCsUA87dUnqpKhQr9upS1JHRYV6zTF1SeqoqFCvO/tFkjoqKtRrrtIoSR0VFep1V2mUpI6KCvWaa79IUkdFhbqduiR1VlSoV3c+muh1GZLUt4oLdRt1SWqvuFC3U5ek9ooLdTNdktorK9TDTl2SOikr1IeCcTNdktoqK9QjGLdTl6S2ygr1WriglyR1YKhL0gApL9RdpVGS2iov1O3UJamtskI9DHVJ6qSsUG8sE5AOwUhSS8WFOnhLO0lqp8hQd011SWqtyFCfcPhFklrqKtQj4qqIeDQiNkfE1S323xART0TEYxFxyeyXWak7/CJJHdVnOiAilgA3ARcD48CDEXF7Zu5s7L8aWAecBRwC5s9VsbUw1CWpk2469fXAPZm5PTNHgbuBS5v2/xbw7zPzYFYOzUWhAPUhQ12SOukm1E8BtjY93wacDBAR84BVwK9HxCMRcVtELJ/6AhFxTURsjIiNO3fufPHF2qlLUkfdhPow0Lw04gTVMAzACmAZVfd+DvA4cP3UF8jMmzNzXWauGxkZedHFHh1T90SpJLXUTajvAFY3PV8DPNF4/DSwLzPvyuqKoL8Bzp7dEp9Xm5zSOG6oS1Ir3YT6ncD6iFgZEauASxrbyMwjwNcj4k2NY68A7p+TSnm+U3dKoyS1NmOoN06OXg/cC3wNuA64LCLe0zjk3wDvj4jNVGPtH56jWr34SJJmMOOURoDM3ABsaLNvC/BTs1dSe0cvPjLUJamlsq4oDTt1SeqkrFD3ilJJ6shQl6QBUmaoO/tFkloqM9Tt1CWpJUNdkgZIWaHu2i+S1FFRoe4qjZLUWVGh7iqNktRZUaFer1XlGuqS1FpRod7IdK8olaQ2igr1yU7dVRolqbWiQn3ITl2SOioq1CdPlLpKoyS1VlSoTw6/2KlLUmtFhfrkiVI7dUlqrahQt1OXpM6KCvXJTt1VGiWptaJC/ejFR+MTPa5EkvpTUaF+dEEvG3VJaqmsUD+6oJeduiS1UlaoH13Qq8eFSFKfKivUa3bqktRJoaHe40IkqU8VFeqNTHdKoyS1UVSoRwRDtXD4RZLaKCrUgUao97oKSepP5YV62KlLUjvFhXrdTl2S2iou1GuOqUtSW8WFer0Wzn6RpDaKC/WqUzfUJamV4kK9bqhLUlvFhXotwptkSFIbxYV6fSi8nZ0ktVFcqA/ZqUtSW+WFei2YcPaLJLVUZKiPeesjSWqpq1CPiKsi4tGI2BwRV7c55k8iYvPsljednboktVef6YCIWALcBFwMjAMPRsTtmbmz6ZifAVbNWZVNhmqOqUtSO9106uuBezJze2aOAncDl07ujIgTgA8C75+bEl9oyHnqktRWN6F+CrC16fk24OSm5x8APgk80+4FIuKaiNgYERt37tzZ7rCuVKs0GuqS1Eo3oT4MNK+gNUE1DENEnA9ckJm3dnqBzLw5M9dl5rqRkZEXXSzYqUtSJ92E+g5gddPzNcATjcfvBM6MiAeBLwKnRMRnZrfEFzLUJam9bkL9TmB9RKyMiFXAJY1tZOZ7MvPszLwQuBx4IjPfNnflNkLd2S+S1NKMs18yczQirgfubWy6DrgsIs7IzBvntLoW7NQlqb0ZQx0gMzcAG2Y45jHgzJdc0Qw8USpJ7RV5RamhLkmtGeqSNEAMdUkaIGWGurNfJKmlIkPdVRolqbXyQj1cpVGS2iku1OtDrtIoSe0UF+q18B6lktROcaFedz11SWqruFCv1ezUJamd4kLdTl2S2isu1GvOU5ektooL9brDL5LUVnGhPhQOv0hSO+WFeq0q2W5dkqYrMNSrr3brkjRdgaHe6NQ9WSpJ0xQY6tVXO3VJmq7AUK9Kdk11SZquvFCP6quhLknTlRfqQ3bqktROeaEeVatuqEvSdMWFer3WCHVnv0jSNMWFem0y1L2lnSRNU1yo26lLUnvFhfrRTn1ioseVSFL/KS7Uj3bqZrokTVNcqNcas1/G7NQlaZriQn2yUzfTJWm64kJ9qGanLkntFBvqrtIoSdMVG+pjzlOXpGmKC/XJE6XOU5ek6YoL9fqQa79IUjvFhXrNBb0kqa3iQv35i48MdUmaqrhQHzLUJaktQ12SBkhXoR4RV0XEoxGxOSKunrLv2oj4dkRsjYhPR0R9bkqtDNerkg+PefGRJE01Y6hHxBLgJuC1jV8fioiRpkP2AhcAZwAnAW+bgzqPWvWyEwB48tmDc/k2klSkbjr19cA9mbk9M0eBu4FLJ3dm5qcy87nMHAMeAl4+N6VWFs2vs3ThPLbvNtQlaapuQv0UYGvT823AyVMPioiFwM8Dt7fYd01EbIyIjTt37nyxtR61eukCthnqkjRNN6E+DDQPYE8A480HREQN+HPg45n52NQXyMybM3NdZq4bGRmZuvtHtmbZArb/0FCXpKm6CfUdwOqm52uAJyafREQAtwAPZ+YnZ7e81lYvXcj23QdJlwqQpBfoJtTvBNZHxMqIWAVc0tg26Q+A0cz8wFwU2MrqZQs4eGSc3QeOHKu3lKQizDj9MDNHI+J64N7GpuuAyyLiDOA+4BpgS0Rc1dj//sz8yzmptmH10gUAbNt9gJcvGp7Lt5KkonQ1pzwzNwAb2uw+5hcwrVlWhfr23Qf5iTVLj/XbS1LfKu6KUmgKdU+WStILFBnqJy6Yx6LhIac1StIURYZ6RLDaaY2SNE2RoQ6wZtlCO3VJmqLYUF+9dAHbdx/odRmS1FfKDfVlC9hzaIy9h5yrLkmTyg31pc6AkaSpig31yWmN254x1CVpUrGhfvqKxQzVgvsfe6bXpUhS3yg21E9cOI/XnTXC3zz4pLe2k6SGYkMd4C0XrWZ0zyG+vmVXr0uRpL5QdKi/ce1JLJ5f57YHtve6FEnqC0WH+gnzhvi581Zxxz+McvC58Zl/gyQNuKJDHaohmH2Hx/jf/7ij16VIUs8VH+oXn76cs05azMe/vJmx8YmZf4MkDbDiQ71WC967/hy2PL2fz27a1utyJKmnig91gDe8aiWvPnUpH/3S9zh0xLF1ScevgQj1iOC333QOo3sO8SdffbTX5UhSzwxEqEM1tn75+av4vS9/j+/v3NfrciSpJwYm1AFuuPJcFswb4j9+9iEmvMpU0nFooEJ95ZIT+MAVa9m4dTef+prDMJKOPwMV6gBvffVq3rj2JH73ju+waauLfUk6vgxcqEcEN/7iBbxi6QLefesD7Np3uNclSdIxM3ChDnDignl88l++mmcOPMc1n97kNEdJx42BDHWA81afyEffdiHffHw3//YvHvBqU0nHhYENdYDLzz+Z/3LluXzp2z/gvZ99yGCXNPDqvS5grv3qPzuNvYfG+PDfPcKhI+P83tsvYrg+0D/LJB3Hjot0e/fPnMn7r1jLHf8wyr/602+we/9zvS5JkubEcRHqAL/+2lfykasuYOPW3Vz5+1/l2zv29LokSZp1x02oA7z11Wv4zDUXc/jIBG/+xNe45e+3eOWppIFyXIU6wEWnLuOL1/40rzt7hA9+4dtc9Uf38vCTdu2SBsNxF+oAKxbP5+Z3vIYP/4ufYMvT+7ni43/P9bd9i9FnD/W6NEl6SSLz2A4/rFu3Ljdu3HhM37OTZw8c4SN3PcJffONxIoK3rTuFX/up0zh9ZHGvS5OkoyJiU2aum/G44z3UJz3xzAE+cfdmbntgO8+NT/DTP76CKy94BZedu4oTF8zrdXmSjnOG+ov01N5D3Hrf43zum9vYtvsgw0M1Xnf2CJefv4pLzljBSS87odclSjoOGeovUWby4BM/5Pb/t4O/fehJntpbLQx2+opF/OTpy/mnr1zG2pNP5PSRRcwbOi5PTUg6hgz1WTQ+kTz85B7u27KL+7bs4huPPsPew2MADA/VOHPlYs5ZtYRTly/klGULOXX5QtYsW8CKxfMNfEmzwlCfQ+MTyean9vGd0T08vGMP39mxl+/+YC+jew4x9eNcunAeyxcNs3zxfFYsHmb5ovksPqHO4vl1Fg4PsWh+nUXDdRbNf/7xcL3G8FCNefWgXqse14eCeUM15g0FEdGbP7iknuk21Lta+yUirgL+KzAOfCgzP9W07zzgVmAp8L+AazNzoFfOGqoFZ69awtmrlvDmC1cf3X54bJztuw/y+DMH2Lb7ILv2Pceu/Yd5et9hnt73HI+M7mXX/l3sOzTG2Eu46KleqwJ+qBZEQC2avlKtKV8Ljm6rNX4I1GrPH1OLgG5+NnRRZjd/km6bh+5eq5vX6f+Lyo5xP/WilVJnCS591Up+583nzel7zBjqEbEEuAm4mCrUH4yI2zNzZ+OQTwL/CbgTuBu4Evj83JTb3+bXhzh9ZHFX0yEPj41z4PA4+w6PceC56uv+w2MceG6Mw2MTHBlPxsYnODJePT7S4vH4RBVemTCRz3+tfl4kE439E43tNO1Pqsfd5Ho3/zPo7nW6OKjr15qdmvpCAYVGCUXS/fdYr/z4SUvm/D266dTXA/dk5naAiLgbuBT4q4gYAV6ZmXc09t0KvInjNNR/FPPrQ8yvD7Fs0XCvS5E0QLo5i3cKsLXp+Tbg5MbjNcDjbfYdFRHXRMTGiNi4c+fOqbslSbOkm1AfBprHyCeohmFm2ndUZt6cmesyc93IyMiLrVWSNINuQn0HsLrp+RrgiS72SZKOsW5C/U5gfUSsjIhVwCWNbWTm48D+iHh9RAwB7wD+x5xVK0nqaMYTpZk5GhHXA/c2Nl0HXBYRZ2TmjcA7gT+jmtK4ITO/OmfVSpI66mqeemZuADa02fdN4PzZK0mS9GJ5DbskDRBDXZIGyDFf+yUidvLCee8/qhXA07NUzlzo9/rAGmeLNc4Oa+zOj2XmjHPCj3mov1QRsbGbRW16pd/rA2ucLdY4O6xxdjn8IkkDxFCXpAFSYqjf3OsCZtDv9YE1zhZrnB3WOIuKG1OXJLVXYqcuSWrDUJcGQEQsiIizel1HJ9Z4bBQT6hFxVUQ8GhGbI+LqXtcDEBEnRMTNEfHdiNgaEb/V2H5tRDweEY9ExM/1uk6AiBiOiIcj4pbG876qMSJOjIi/iojtEfH9Rr39VuN/iIjvNb4P393Y1tMaI+JlEfF54AfAbzdtb1lXRPxuRGyLiG9FxGt6VWNELI+IzzQ+z+9HxNv7rcamfcsj4gcR8b5e1ti1zOz7X8ASqiV9VwOrgFFgpA/qWg78AtUNyVZQfUO8Dvhuo+a1wJPAvD6o9Qbgi8AtwBn9ViPw58D7Gp/lCf1WI3Aa8BiwqPH3/ixwbq9rBBZT3YnsN4BbGttafnbAzwJfpVrz6Y3Agz2s8Rzg9Y3HZwI/7Lcam/b9KXAH8L7G857U2O2vUjr1o7fUy8xRqnuhXtrjmsjMXZn5uaw8TfWD558Df52ZezPzYaog6OlP8oh4FfBPgL9ubHoLfVRj05LOH2p8lof6rUbgSOPrBNU/5r3A5fS4xszcl5lfBsaaNrf77N5KtZLqWGbeBYw0PvtjXmNmficzv9J4vJnq813QTzUCRMQbGtu+3rS5JzV2q5RQ73RLvb4QEedRdZgr6KNao7pD88eAa5s299vneS7wKPC5xnDBjfRZjVndo/cG4D7gS8AvU90Upm9qbNLus5u6fTt9UG9jeOibmbmHPqoxIhYAv8OU4Rj6qMZWulp6tw90ddu8XomIFcCngV8Drqa/av3XwFcyc3NEvLaxrd8+z5VUwwQ/CeymCs1VwENNx/S0xoh4GVWQXwu8EngP1T/mfvocJ7X7++23v3ci4kzgw8AVjU39VOMNwO9n5u6qNzqqn2qcppRQ3wG8vun5Gl7436GeiYhlwN8C/zkz7290Hf10i793AEsi4heBl1ONCX+M/qrxKWBTZm4DiIi7qP6R9FONvwI81Bgy+EpEvIXq3E4/1Tip3W0mp25/BVUX3xMR8WPAZ4FfzczHGpv7qcZfprrr23upmoyMiEfprxqnKWX4pe0t9Xqp0b3dDnwwM+9obP4C8PaIWBgRa6mC9MFe1ZiZl2Tm+Zl5IfAB4DaqH0J9UyPVkMbaiHhFRMwH3gDs67MaDwEXRsS8iFgCnEX1P4p+qnFSu+/BLwDvjIihiHgj8N3MfKYXBUbEauB/Au/K6kY7k/qmxsw8JTMvbPzb+UOqrv3WfqqxlSI69WxxS73M3N/Lmhp+E7gI+GhEfLSx7TLgvwP/SBUEv5GNU+b9IjM3RUTf1JiZ+yPi3wF3AfOpTkLd1Aj4vqiR6u/0Z4EtwEHgzzLza73+HBs/YB6gmulyQkS8HngXLb4HI+I2qtlZW4BdVJ1or2qcnDH2l01DG2upmo5+qfFdmfl/Whzekxq75TIBkjRAShl+kSR1wVCXpAFiqEvSADHUJWmAGOqSNEAMdUkaIIa6JA0QQ12SBoihLkkD5P8DWGgyuFETV5IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(val_losses);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# torchviz\n",
    "\n",
    "https://github.com/szagoruyko/pytorchviz/\n",
    "\n",
    "> conda install graphviz\n",
    "\n",
    "> pip install torchviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-20T16:42:54.140000Z",
     "start_time": "2019-06-20T16:42:53.646544Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"263pt\" height=\"300pt\"\n",
       " viewBox=\"0.00 0.00 262.82 300.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 296)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-296 258.8154,-296 258.8154,4 -4,4\"/>\n",
       "<!-- 4897175592 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>4897175592</title>\n",
       "<polygon fill=\"#caff70\" stroke=\"#000000\" points=\"169.8066,-20 65.8496,-20 65.8496,0 169.8066,0 169.8066,-20\"/>\n",
       "<text text-anchor=\"middle\" x=\"117.8281\" y=\"-6.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddmmBackward</text>\n",
       "</g>\n",
       "<!-- 4897175984 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>4897175984</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"55.4854,-88 .1708,-88 .1708,-56 55.4854,-56 55.4854,-88\"/>\n",
       "<text text-anchor=\"middle\" x=\"27.8281\" y=\"-74.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">W1.bias</text>\n",
       "<text text-anchor=\"middle\" x=\"27.8281\" y=\"-62.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1)</text>\n",
       "</g>\n",
       "<!-- 4897175984&#45;&gt;4897175592 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>4897175984&#45;&gt;4897175592</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M51.4746,-55.7102C64.7617,-46.5569 81.2823,-35.176 94.5149,-26.0603\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"96.8209,-28.7218 103.0704,-20.1664 92.8497,-22.9573 96.8209,-28.7218\"/>\n",
       "</g>\n",
       "<!-- 4897175368 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>4897175368</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"162.2922,-82 73.364,-82 73.364,-62 162.2922,-62 162.2922,-82\"/>\n",
       "<text text-anchor=\"middle\" x=\"117.8281\" y=\"-68.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">TanhBackward</text>\n",
       "</g>\n",
       "<!-- 4897175368&#45;&gt;4897175592 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>4897175368&#45;&gt;4897175592</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M117.8281,-61.762C117.8281,-53.185 117.8281,-40.6836 117.8281,-30.1154\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"121.3282,-30.0475 117.8281,-20.0475 114.3282,-30.0476 121.3282,-30.0475\"/>\n",
       "</g>\n",
       "<!-- 4897215208 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>4897215208</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"167.8066,-150 63.8496,-150 63.8496,-130 167.8066,-130 167.8066,-150\"/>\n",
       "<text text-anchor=\"middle\" x=\"115.8281\" y=\"-136.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">AddmmBackward</text>\n",
       "</g>\n",
       "<!-- 4897215208&#45;&gt;4897175368 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>4897215208&#45;&gt;4897175368</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M116.1232,-129.9664C116.4127,-120.1231 116.8626,-104.827 117.2276,-92.4189\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"120.7362,-92.1734 117.5318,-82.0748 113.7392,-91.9675 120.7362,-92.1734\"/>\n",
       "</g>\n",
       "<!-- 4897265368 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4897265368</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"102.4854,-224 47.1708,-224 47.1708,-192 102.4854,-192 102.4854,-224\"/>\n",
       "<text text-anchor=\"middle\" x=\"74.8281\" y=\"-210.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">W0.bias</text>\n",
       "<text text-anchor=\"middle\" x=\"74.8281\" y=\"-198.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (16)</text>\n",
       "</g>\n",
       "<!-- 4897265368&#45;&gt;4897215208 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>4897265368&#45;&gt;4897215208</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M84.5446,-191.8849C90.4912,-182.0222 98.1303,-169.3524 104.3711,-159.0019\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"107.4544,-160.6665 109.6206,-150.2954 101.4597,-157.052 107.4544,-160.6665\"/>\n",
       "</g>\n",
       "<!-- 4897266040 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>4897266040</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"192.8027,-218 120.8535,-218 120.8535,-198 192.8027,-198 192.8027,-218\"/>\n",
       "<text text-anchor=\"middle\" x=\"156.8281\" y=\"-204.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">TBackward</text>\n",
       "</g>\n",
       "<!-- 4897266040&#45;&gt;4897215208 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4897266040&#45;&gt;4897215208</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M150.7784,-197.9664C144.6013,-187.7213 134.8626,-171.5693 127.2325,-158.9145\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"130.0635,-156.8314 121.9026,-150.0748 124.0688,-160.4458 130.0635,-156.8314\"/>\n",
       "</g>\n",
       "<!-- 4897266152 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>4897266152</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"191.3145,-292 122.3418,-292 122.3418,-260 191.3145,-260 191.3145,-292\"/>\n",
       "<text text-anchor=\"middle\" x=\"156.8281\" y=\"-278.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">W0.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"156.8281\" y=\"-266.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (16, 8)</text>\n",
       "</g>\n",
       "<!-- 4897266152&#45;&gt;4897266040 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4897266152&#45;&gt;4897266040</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M156.8281,-259.8849C156.8281,-250.5254 156.8281,-238.6379 156.8281,-228.6036\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"160.3282,-228.2954 156.8281,-218.2954 153.3282,-228.2954 160.3282,-228.2954\"/>\n",
       "</g>\n",
       "<!-- 4897176824 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>4897176824</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"254.8027,-82 182.8535,-82 182.8535,-62 254.8027,-62 254.8027,-82\"/>\n",
       "<text text-anchor=\"middle\" x=\"218.8281\" y=\"-68.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">TBackward</text>\n",
       "</g>\n",
       "<!-- 4897176824&#45;&gt;4897175592 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>4897176824&#45;&gt;4897175592</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M202.15,-61.762C186.0452,-51.8758 161.4471,-36.776 142.9677,-25.4322\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"144.5493,-22.2963 134.1959,-20.0475 140.8872,-28.262 144.5493,-22.2963\"/>\n",
       "</g>\n",
       "<!-- 4897215096 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>4897215096</title>\n",
       "<polygon fill=\"#add8e6\" stroke=\"#000000\" points=\"254.3145,-156 185.3418,-156 185.3418,-124 254.3145,-124 254.3145,-156\"/>\n",
       "<text text-anchor=\"middle\" x=\"219.8281\" y=\"-142.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\">W1.weight</text>\n",
       "<text text-anchor=\"middle\" x=\"219.8281\" y=\"-130.4\" font-family=\"Times,serif\" font-size=\"12.00\" fill=\"#000000\"> (1, 16)</text>\n",
       "</g>\n",
       "<!-- 4897215096&#45;&gt;4897176824 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>4897215096&#45;&gt;4897176824</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M219.5911,-123.8849C219.4535,-114.5254 219.2787,-102.6379 219.1311,-92.6036\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"222.6263,-92.2428 218.9795,-82.2954 215.627,-92.3458 222.6263,-92.2428\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x123e45f98>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torchviz import make_dot, make_dot_from_trace\n",
    "\n",
    "model = nn.Sequential()\n",
    "model.add_module('W0', nn.Linear(8, 16))\n",
    "model.add_module('tanh', nn.Tanh())\n",
    "model.add_module('W1', nn.Linear(16, 1))\n",
    "\n",
    "x = torch.randn(1,8)\n",
    "\n",
    "make_dot(model(x), params=dict(model.named_parameters()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<!--NAVIGATION-->\n",
    "< [In-Depth: Decision Trees and Random Forests](09.08-Random-Forests.ipynb) | [Contents](Index.ipynb) |[In-Depth: Neural Network Advanced](09.10.neural_network_advanced.ipynb)>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "170px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
